Detection method, device and storage medium based on neural network chip performance

Abstract

The present application is applicable to the field of neural network technology, and provides a detection method, device and storage medium based on the performance of a neural network chip. The detection method includes: sparsely processing the weights of a neural network model through binary masks and importance scores to reduce the memory requirements of the neural network algorithm at runtime, so that the optimized neural network algorithm can be deployed in performance-constrained devices.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a detection method, device and storage medium based on neural network chip performance. Background Art

[0002] Artificial Intelligence (AI) chips, also known as AI accelerators or computing cards, are modules specifically designed to handle large amounts of computing tasks in AI applications. AI chips can apply a variety of deep algorithms to enable AI chips to be applied in multiple fields.

[0003] However, the huge size of the parameters associated with deep learning algorithms running in AI chips (for example, neural network algorithms for driver fatigue detection) and their high power and computing power requirements have severely limited the use of deep learning algorithms in resource-constrained embedded systems. Summary of the invention

[0004] The present application provides a detection method, device and storage medium based on the performance of a neural network chip. The weight matrix of an optimized neural network algorithm is sparsely processed to reduce the memory requirement of the neural network algorithm at runtime, so that the optimized neural network algorithm can be deployed in a performance-constrained device.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a detection method is provided, the detection method comprising:

[0007] Get the weight matrix corresponding to the neural network model;

[0008] The following objective function is constructed based on the weight matrix:

[0009]

[0010] Where W is the weight matrix, L is the cross entropy loss function, ⊙ is the Hadamard product, M * (S) is the binary mask after the importance score is approximately binarized, ‖·‖0 is the norm of the vector L0, which is used to represent the number of non-zero elements in the vector; k is the network sparsity, S is the importance score vector, M is a preset binary mask;

[0011] Performing sparse processing on the weight matrix according to the objective function to obtain sparse weights;

[0012] The sparse weights are loaded into the neural network model, and image detection is performed based on the neural network model.

[0013] In a possible implementation manner of the first aspect, performing sparse processing on the weight matrix according to the objective function to obtain the sparse weights includes:

[0014] The following transition function is determined according to the objective function:

[0015]

[0016] Among them, m i represents the i-th element in the binary mask vector M, s i is the i-th element in the importance score vector S; is the kth largest element in S;

[0017] The weight matrix is ​​subjected to sparse processing according to the transition function to obtain sparse weights.

[0018] In a possible implementation of the first aspect, performing sparse processing on the weight matrix according to the transition function to obtain the sparse weights includes:

[0019] Determine the initial importance score vector;

[0020] Determining an initialized binary mask according to the initialized importance score vector;

[0021] Determining initialized sparse weights according to the initialized binary mask and the weight matrix;

[0022] Determining the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask according to the initialized sparsification weight;

[0023] Updating the initialized importance score vector according to the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask;

[0024] Then, a preset number of loop iterations of the binary mask and the importance score vector are performed to determine a sparse binary mask, and the sparse weight is determined according to the sparse binary mask.

[0025] In a possible implementation of the first aspect, after the sparse weights are loaded into the neural network model, the detection method further includes:

[0026] Performing performance testing on the neural network model to determine performance indicators of the neural testing model;

[0027] The initialization parameters corresponding to the loop iteration are updated according to the performance indicator, and the initialization parameters include iteration period, sparsity and learning rate.

[0028] In a possible implementation of the first aspect, the gradient of the objective function with respect to the importance score is

[0029]

[0030] The gradient of the cross entropy loss function with respect to the binary mask is

[0031] The updating of the initialized importance score vector according to the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask comprises:

[0032] If the gradient of the objective function with respect to the importance score is equal to the gradient of the cross entropy loss function with respect to the binary mask, the initialized importance score vector is updated according to the following formula:

[0033] Among them, t≥C, C is the iteration period, S t is the current importance score vector, S t+1 is the updated importance score vector.

[0034] In a possible implementation of the first aspect, the performance indicator includes a plurality of test indicators;

[0035] The updating of the initialization parameters corresponding to the loop iteration according to the performance indicator comprises:

[0036] Obtaining the weight corresponding to the test indicator;

[0037] Determining the current comprehensive index according to the weight and the value of the test index;

[0038] The initialization parameters corresponding to the loop iteration are updated according to the comprehensive index.

[0039] In a possible implementation of the first aspect, updating the initialization parameter corresponding to the loop iteration according to the current comprehensive indicator includes:

[0040] k=k+δ*(target_SI-current_SI);

[0041] α=α*(target_SI / current_SI)^β;

[0042] C=C+γ*(target_SI-current_SI);

[0043] Among them, C is the iteration period, k is the sparsity, α is the learning rate, target_SI is the value of the desired comprehensive index, current_SI is the value of the current comprehensive index, δ is the adjustment sparsity of the sparsity, β is the scaling sparsity of the learning rate, and γ is the adjustment sparsity of the iteration period.

[0044] In a possible implementation of the first aspect, the neural network model is a multi-task convolutional neural network model.

[0045] In a second aspect, a detection device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the detection method in any optional implementation of the first aspect are implemented.

[0046] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the detection method as described in any one of the first aspects are implemented.

[0047] The detection method, device and storage medium provided in the embodiments of the present application have the following effective effects:

[0048] The detection method provided in the present application performs sparse processing on the weight matrix of the neural network through a binary mask, and loads the sparsely processed weights into the neural network; by pruning the weights of the neural network, the requirements for device performance when the neural network model is running are reduced, so that the neural network-based detection method can be applied to performance-limited devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a flow chart of a method for detecting performance of a neural network chip based on an embodiment of the present application;

[0050] Figure 2 A schematic flow chart of another method for detecting performance of a neural network chip provided in an embodiment of the present application;

[0051] Figure 3 A schematic flow chart of another method for detecting performance of a neural network chip provided in an embodiment of the present application;

[0052] Figure 4 A schematic flow chart of another method for detecting performance of a neural network chip provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] It should be noted that the terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, "multiple" refers to two or more than two, and "at least one", "one or more" refers to one, two or more. The terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, it is defined that the "first" and "second" features can explicitly or implicitly include one or more of the features.

[0055] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0056] To facilitate understanding, the terms involved in the embodiments of the present application are first briefly introduced.

[0057] 1. Multi-task Cascaded Convolutional Networks (MT-CNN)

[0058] MT-CNN is a neural network model for face detection tasks, which can integrate face detection and facial key point detection in the same model.

[0059] The MTCNN network structure is a three-level cascade network, which can be generally divided into three layers of network structures: P-Net, R-Net, and O-Net. The model adopts the idea of ​​candidate box plus classifier, which can take into account both speed and accuracy to achieve fast and efficient face detection.

[0060] 2. Weight pruning

[0061] Weight pruning is a model optimization technique. In weight pruning, it gradually zeroes the model weights during training to achieve model sparsity.

[0062] In order to ensure the safety of vehicle driving and reduce the occurrence of vehicle accidents, vehicles are often equipped with driver fatigue detection equipment to detect whether the driver is driving fatigued. The fatigue driving detection methods running in existing fatigue driving detection equipment are mostly designed based on a complete deep neural network algorithm model, which has high energy consumption. In addition, due to the limited computing performance of most on-board equipment, the detection and reasoning speed during use is slow, and it is impossible to detect the driver's status in real time.

[0063] Based on the above problems, an embodiment of the present application provides a detection method based on the performance of a neural network chip. The detection method performs sparse processing on the weight matrix of the neural network through a binary mask, and loads the sparsely processed weights into the neural network; by pruning the weights of the neural network, the requirements for device performance when the neural network model is running are reduced, so that the neural network-based detection method can be applied to performance-limited devices.

[0064] See also Figure 1 , Figure 1 A flow chart of a method for detecting the performance of a neural network chip provided in an embodiment of the present application, the method comprising the following steps:

[0065] S101. Obtain a weight matrix of a neural network.

[0066] Among them, the weight matrix can be the initial weight matrix of the image neural network, that is, after determining the application scenario of the detection method (such as fatigue driving detection), by collecting image data for training, a neural network weight matrix that can be used for detection and meets the preset requirements is obtained, that is, the original weight matrix.

[0067] S102, performing sparse processing on the weight matrix through binary mask and importance score to obtain sparse weights.

[0068] Specifically, the weight matrix of the neural network is sparsely processed through binary masks and importance scores to achieve weight pruning of the neural network.

[0069] In some embodiments, step S102 includes:

[0070] The objective function is established based on the weight matrix, and the weight matrix is ​​sparsely processed based on the objective function to obtain the sparse weights, where the objective function is:

[0071]

[0072] Among them, W is the weight matrix, L is the cross entropy loss function, ⊙ is the Hadamard product, M *(S) is the binary mask after the importance score is approximately binarized, M * ∈{0, 1} p ;

[0073] Furthermore, the value of p depends on W.

[0074] Exemplarily, if W is a matrix of a×b×c, then p=a×b×c.

[0075] Among them, ‖·‖0 is the L0 norm of the vector, which is used to indicate the number of non-zero elements in the vector; K is a constant, which is used to indicate the sparsity of the network and is used to determine how many remaining weights are retained in the original network; S is the importance score, which is used to determine which elements in the matrix W will be pruned; M is a pre-set binary mask, M∈{0,1} p .

[0076] In this way, the objective function is constructed through the binary mask, importance score and weight matrix. The objective function is a training loss model based on the importance score and the binary mask. The optimal binary mask is determined by the training loss model, and then the sparsification weight is determined based on the optimal binary mask, thereby realizing the sparse processing of the weight matrix of the neural network.

[0077] The above objective function uses the Hadamard product, so W, S, M, M * The form should be consistent. With the three-dimensional matrix X∈{0,1} a×b×c Perform Hadamard product, X is a three-dimensional matrix of a×b×c composed of 0 and 1. If X ijk = 1, then the element in W corresponding to the position of X remains unchanged after multiplication. ijk = 0, then the multiplication of the elements in W corresponding to the X position is 0. The above objective function is realized by the above idea between the weights, that is, through the three-dimensional matrix Complete the pruning of the weight W block.

[0078] In order to obtain the optimal three-dimensional matrix X (i.e., the optimal binary mask mentioned above), each element in W has its importance score. The elements with large importance scores are retained and multiplied by 1, and the elements with small importance scores are pruned, that is, they are multiplied with 0 at the corresponding position to achieve pruning.

[0079] It can be understood that the solution of the above objective equation can be regarded as a two-level optimization problem and solved by the alternating optimization method: first, the optimal binary mask is solved based on the current importance score, that is, the constraint formula M * (S), and then use the solved optimal binary mask to update the importance score of the next iteration. The final optimal binary mask can be determined through repeated iterations.

[0080] Based on the above solution ideas, the specific process of sparse binarization of importance scores is first modeled as a binary programming problem, that is, the objective function is expanded as:

[0081]

[0082] In formula 1, is a variable that changes with the optimization of the importance score, and is a constant k. Therefore, the optimal solution of Formula 1 is equivalent to Formula 2, that is, in order to make Minimum, then S T Mmax:

[0083]

[0084] In formula 2, in order to obtain M * (S), which is equivalent to solving To find the maximum value, first take out the index of the largest k elements in the importance score vector S. Then use the mask M * (S) The value of the element under the corresponding index is set to 1, and the rest are set to 0.

[0085] In this way, the optimal binarization function M is obtained based on the importance score. * (S) is as follows:

[0086]

[0087] Among them, m i is the i-th element in the binary mask vector M, s i is the i-th element in the importance score vector S; is the kth largest element in the importance score vector S.

[0088] From formula 3, we can know that the purpose of the optimal binarization function of importance score is to retain the largest k elements in the importance score vector based on the importance score size. By inputting the importance score into the binarization function, an approximate binary mask can be mapped. Therefore, the objective function is equivalent to the following transition function:

[0089]

[0090] So far, the objective function and the optimal sparse binarization function of importance score, namely the transition function, have been obtained. By solving the transition function, the optimal binary mask can be determined, and the optimal sparse network can be obtained based on the optimal binary mask.

[0091] In the transition function, since M and S exist at the same time, one of them can be used to solve the other. The solution can be solved by alternating optimization: first, based on the optimal binarization function, the optimal binary mask based on the current importance score can be obtained. Then the importance score is optimized, and the optimal binary mask cuts out a sparse network and then back-propagates.

[0092] In some embodiments, the weight matrix is ​​subjected to sparse processing according to the transition function, and the sparse weights obtained include:

[0093] Determine the initial importance score vector;

[0094] Determine an initialized binary mask according to the initialized importance score vector and the transition function;

[0095] Determine the initialized sparse weights based on the initialized binary mask and weight matrix;

[0096] Determine the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask based on the initialized sparsified weights;

[0097] Update the initialized importance score vector according to the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask;

[0098] Then, a preset number of loop iterations of the binary mask and the importance score vector are performed to determine a sparse binary mask, and the sparse weight is determined according to the sparse binary mask.

[0099] Since M and S exist at the same time in the transition function, one of M and S can be used to solve the other. The initialized importance score vector is determined, and then the corresponding binary mask is determined based on the initialized importance score vector. Then, the corresponding sparse weight is determined based on the initialized binary mask, and the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask are determined based on the sparse weight. The importance score vector is updated by the gradient descent method, and then the cycle is repeated until a binary mask that meets the requirements is determined, and the corresponding sparse weight is determined based on the binary mask.

[0100] For ease of understanding, the following is an exemplary explanation of determining the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask according to the initialized sparsified weights; and updating the initialized importance score vector according to the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask:

[0101] First, determine the gradient g of the objective function with respect to the importance score S, as shown below:

[0102]

[0103] Then, determine the gradient g of the cross entropy loss function with respect to the binary mask M , as shown below:

[0104]

[0105] Then, the gradient is approximated, that is, the gradient of the weight score g S Approximately equal to the gradient g of the cross entropy loss function with respect to the binary mask M , as shown below:

[0106] g S ≈g M (Formula 6);

[0107] Then, the sparse network obtained by the binary mask is back-propagated and the importance score is updated using the approximate gradient, i.e.

[0108] Then use Formula 6 to backpropagate this sparse network and calculate the gradient of the loss function with respect to the binary mask M0:

[0109]

[0110] According to Formula 7, the gradient of the importance score is approximately equal to the gradient of the loss function with respect to the binary mask, as shown below:

[0111]

[0112] Then use gradient descent to update S0 as follows:

[0113]

[0114] Among them, α is the learning rate. During the back propagation process, the weight matrix does not participate in the update. The above is a complete iterative process. After the preset number of iterations, the importance score S and the binary mask M are optimized alternately. Finally, the final importance score S can be obtained. * , and input it into the binarization function to obtain the optimal binary mask M * (S). The optimal binary mask M * (S) Applied to the original randomly initialized network, a sparse network can be pruned from it, that is, the sparse weights.

[0115] Among them, Formula 9 is to determine the updated importance score vector based on the importance score vector to be initialized (i.e., the first iteration process). If the subsequent importance scores can be calculated using the following formula:

[0116] Among them, t≥C, C is the iteration period, S t is the current importance score vector, S t+1 is the updated importance score vector.

[0117] In summary, in the above process, the importance score S0 is first initialized; then S0 is input into M * (S), and get the binary mask M0. Then multiply M0 by the weight matrix, that is, M0⊙W; complete pruning to form a preliminary sparse network, and then determine the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask based on the initial sparse network, and update the corresponding importance score based on the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask, and then cycle in sequence until the corresponding optimal binary mask is determined, and determine the sparse weight based on the optimal binary mask.

[0118] In this way, the weight matrix is ​​sparsely processed based on the objective function, and the sparse weights are obtained, which includes the following steps:

[0119] Step 1: Determine the cross entropy loss function L, sparsity k, cycle C, learning rate α, and weight matrix W;

[0120] Step 2, determine the initialized importance score vector S0;

[0121] Step 3: Set the t value, where 0≤t≤C;

[0122] Step 4: Calculation ‖M‖0≤k is assigned to M t ;

[0123] Step 5: Multiply the binary mask by the weight matrix to complete pruning.

[0124] Step 6: Calculate the gradient of the cross entropy loss function with respect to the binary mask

[0125] Step 7: Calculate the gradient S of the objective function with respect to the importance score t ,

[0126] Step 8: Update the gradient. Assign to S t+1 ;

[0127] Step 9, then loop through steps 4 to 8;

[0128] Step 10: The optimal binary mask M obtained through final optimization* , according to the optimal binary mask M * Get the optimal sparse network M * ⊙W.

[0129] In this way, by looping through the above steps, multiple iterations are completed to determine the optimal binary mask M * , and then according to the optimal binary mask M * Get the optimal sparse network M * ⊙W, that is, the weight after sparseness. By setting the number of iterations in step 3, the initial importance score S0 is used to obtain a binary mask M at this stage. t , then according to M t Perform weight pruning and obtain the importance score S1 of the next stage according to steps 4 and 5. After completing the preset number of iterations, the sparse weight is obtained: that is, the sparse network M * ⊙W.

[0130] S103, loading the sparse weights into the neural network model, and performing image detection based on the neural network model.

[0131] Specifically, the sparse weights are loaded into the neural network model to achieve weight pruning of the neural network model, and then image recognition is performed based on the neural network model after weight pruning.

[0132] In this way, the weight matrix of the neural network model is sparsely processed through binary masks and importance scores, and the sparsely processed weights are loaded into the neural network; by pruning the weights of the neural network, the requirements for device performance when the neural network model is running can be reduced, so that the neural network-based detection method can be applied to performance-constrained devices.

[0133] In some embodiments, the neural network model is MTCNN. During the image detection process, since the state of the target (e.g., the target person) in the image is random, for example, the driver's posture changes and the surrounding environment (e.g., lighting and occlusion) are random during the driver's driving process. By using a deep hierarchical multi-task MTCNN framework, face detection and alignment are completed simultaneously, and the inherent relationship between the two is used to extract global face features, for example, the positions of the face, left and right eyes, nose, and left and right corners of the mouth can be obtained simultaneously.

[0134] See also Figure 2 , Figure 2 A flowchart of another detection method based on neural network chip performance provided in an embodiment of the present application is provided below. The application of MTCNN in fatigue driving detection is used as an example for explanation. After the sparse weights are loaded into the neural network model, the above detection method further includes:

[0135] S201, input the test data into the neural network model after weight pruning for feature extraction.

[0136] Among them, MTCNN includes three cascaded sub-networks, namely P-Net (proposal network), R-Net (refinement network) and O-Net (output network), which realize the detection of faces and feature point positions from coarse to fine through the three cascaded sub-networks. MTCNN performs face detection through a three-layer cascade network, which simultaneously performs face classification, bounding box regression and feature point positioning. It has good robustness and is suitable for real driving environments.

[0137] Specifically, feature extraction is performed on the test image through a neural network model.

[0138] S202: Classify the extracted features.

[0139] Among them, the test data (such as video data) is input into the neural network model for feature extraction; after feature extraction, the spatial information is adaptively integrated through the generalized mean (GeM) layer of MTCNN to improve the feature mapping capability, the maximum pooling layer can capture the texture information of the image feature map; the average pooling layer can retain the overall content information of the image. The final pooling layer of the traditional convolutional neural network (CNN) chooses to use one of the two, but discards the advantages of the other pooling layer, and cannot capture the discriminative features of a specific field. The MTCNN of this application takes into account the advantages of both: a fine-grained generalized average pooling GeM is introduced in the final pooling layer. The feature map is then converted to a one-dimensional vector in the fully connected layer, and the number of parameters is reduced by random inactivation to prevent the network from overfitting. Finally, the normalized exponential function (softmax) of the neural network model outputs the classification results, such as eyes and mouth.

[0140] S203: Determine the facial features of the driver.

[0141] Among them, facial features include whether the eyes are open or closed, whether the mouth is open or closed, etc.

[0142] It is understandable that when the driver's head moves, if only the information of one eye of the driver is obtained based on the input image, it is easy to misjudge the driver's state. In order to improve the accuracy of recognition, this application extracts the image of the driver's eyes to determine whether the eyes are open or closed. That is, the features obtained by MTCNN are used to determine the positions of the driver's left and right eyes.

[0143] For example, the midpoint of the upper eyelid of the driver's left eye is defined as a1 (x1, y1), the midpoint of the lower eyelid of the left eye is defined as a2 (x2, y2), the midpoint of the upper eyelid of the right eye is defined as a3 (x3, y3), and the midpoint of the lower eyelid of the right eye is defined as a4 (x4, y4). Then, the height of the eye image between the driver's left eye and right eye is defined as h1 and h2. Specifically:

[0144]

[0145] Furthermore, the driver's oral area changes significantly when talking and yawning. The position of the mouth corners is obtained using the same MTCNN as in the previous step. The position of the left mouth corner is b1(x5,y5), the position of the right mouth corner is b2(x6,y6), the position of the upper lip midpoint is b3(x7,y7), and the position of the lower lip midpoint is b4(x8,y8). The width of the mouth image is w3 and the height is h3, specifically:

[0146]

[0147] In this way, the position of the driver's eyes is first determined, and then the relationship between the height values ​​h1 and h2 of the driver's two glasses and the preset height value is determined to determine whether the driver's glasses are closed or open; for example, if h1 and h2 are less than the preset height, it is determined that the driver's glasses are closed;

[0148] It can then be determined whether the driver's mouth is open or closed based on the relationship between the height and width of the mouth;

[0149] For example, if h3>βw3 (β is a constant), it is determined that the driver's mouth is open; otherwise, it is determined that the driver's mouth is closed.

[0150] S204: Determine the driver's fatigue state.

[0151] It is understandable that when a driver enters a fatigue state, there will usually be a series of physiological reactions, such as yawning and closing eyes. Based on the acquisition of multiple states of the eyes and mouth, the driver's fatigue state is evaluated by calculating the eye closure degree y and the mouth opening degree z. The y parameter represents the percentage of eye closure time per unit time, as follows:

[0152]

[0153] Among them, f i The closed frame of the eye, is the number of closed-eye frames per unit time, and N is the total number of frames per unit time. When y is greater than 0.25, the driver is in a closed-eye state for a long time, which can be used as an indicator of fatigue.

[0154] z represents the percentage of oral closure time per unit time, and the specific formula is as follows:

[0155]

[0156] Among them, f j Frames that represent an open mouth, represents the number of frames with the mouth open per unit time, and N is the total number of frames per unit time. When z is greater than 0.5, it can be judged that the driver has been in the state of opening his mouth for a long time, which can also be used as an indicator of fatigue. The larger the values ​​of these two indicators, the higher the degree of fatigue.

[0157] The fatigue state is identified based on the fatigue threshold of y and z. First, the face and feature point positions of the driver's frame image are obtained through MTCNN, and the eye and mouth features are extracted. The state of the eyes and mouth is evaluated. The degree of eye closure and mouth opening of consecutive frame images are calculated, and when the threshold is reached, it is determined that the driver is in a fatigue state.

[0158] S104. Perform performance test on the neural network model.

[0159] Performance testing can help evaluate the processing speed and accuracy of testing equipment in different application scenarios. For example, performance testing of testing equipment running a neural network model can test the performance of the testing equipment when running a neural network model for fatigue detection.

[0160] See also Figure 3 , Figure 3 A flow chart of another method for detecting performance of a neural network chip provided in an embodiment of the present application is provided. After the sparse weights are loaded into the neural network model, the detection method further includes:

[0161] S301: Determine a test set.

[0162] The test set is a credible test set, and the credible test set may be pre-established.

[0163] The test set should contain closed-eye frames and open-mouth frames in various situations, such as daytime, nighttime, highways, and urban areas. In actual road driving, drivers face various situations and different environments, which may affect the generation and detection of closed-eye frames. Therefore, establishing a test set containing various situations can better simulate actual driving scenarios to ensure the reliability and accuracy of the test results. At the same time, the amount of data in the test set should be large enough to ensure the stability and repeatability of the test results.

[0164] S302: Determine the test target and load.

[0165] Before conducting performance testing, you need to clearly define the test objectives and load.

[0166] The test target refers to the test results for the test equipment (such as AI chips), such as the maximum processing speed and highest accuracy of the test equipment. The load refers to the application or test data that needs to be loaded onto the test equipment during the test process. It should be able to simulate real application scenarios and include different data types, sizes, and complexities.

[0167] The test target can be the accuracy of driver fatigue detection, and the load can be the operation of the neural network model and data.

[0168] S303: Determine test indicators.

[0169] Specifically, the performance evaluation of the detection equipment is achieved through test indicators.

[0170] In order to conduct a complete performance test on the testing equipment, various indicators can be comprehensively considered to determine the actual performance and applicable scenarios of the testing equipment. Specifically, the following test indicators can be used:

[0171] Accuracy: Accuracy refers to the proportion of correct predictions made by the detection device on the test data set. For the driver fatigue detection problem, accuracy can be defined as the proportion of correct predictions made by the detection device when predicting whether the driver is in a fatigued state. The higher the accuracy, the better the performance of the detection device.

[0172] Recall: Recall refers to the ability of the detection device to correctly identify the fatigue state. For the driver fatigue detection problem, recall can be defined as the proportion of the detection device correctly predicting that the driver is in a fatigue state. The higher the recall, the better the performance of the detection device.

[0173] Precision: Precision refers to the proportion of samples that are actually fatigued among the samples predicted by the detection device. For the driver fatigue detection problem, precision can be defined as the proportion of the driver correctly predicted by the detection device to be in a fatigued state. The higher the precision, the better the performance of the detection device.

[0174] F1 Score: F1 Score is the weighted average of recall and precision, which can be used as an indicator to comprehensively evaluate the performance of detection equipment. The higher the F1 Score, the better the performance of the detection equipment.

[0175] DRAM access count: The number of times the detection device accesses DRAM. Current computing processors such as CPUs, GPUs, or AI-specific chips are all designed using the von Neumann architecture. 80% of the power consumption occurs in data transmission, and 99% of the time is consumed in the memory writing process, while the energy consumption and time actually used for computing are actually very low. Therefore, the number of DRAM accesses can be used as a reference for testing detection devices used at the edge.

[0176] Inference speed: refers to the speed at which the detection device processes input data and outputs results. This indicator is crucial for real-time applications, especially for applications that require real-time response, such as driver fatigue detection.

[0177] Model size: refers to the size of the space required by the detection device to store the model. Model size is critical to the performance and storage capacity limitations of embedded devices, so the model size should be reduced as much as possible while ensuring high-precision recognition results.

[0178] Energy efficiency ratio: refers to the ratio of the performance of the detection equipment to its energy consumption. Energy efficiency ratio is an important indicator to measure the performance of the detection equipment. The performance and energy efficiency ratio of the detection equipment should be improved as much as possible to achieve efficient algorithm operation and energy-saving application.

[0179] S304. Run performance test.

[0180] Before running a performance test, you need to set up and prepare the test environment. The test environment should be as close to the actual application scenario as possible, including hardware devices, operating systems, and applications. During the test, the test results need to be recorded. Finally, the test results need to be analyzed and optimized to further improve the performance and accuracy of the detection equipment.

[0181] S105. Optimize the neural network model based on the performance test results.

[0182] Specifically, after the test results are determined, the parameters related to weight pruning are adjusted according to the test results to optimize the neural network model.

[0183] In some embodiments, after loading the sparse weights into the neural network model, the detection method further includes:

[0184] Perform performance testing on the neural network model and determine the performance indicators of the neural testing model;

[0185] The initialization parameters corresponding to the loop iteration are updated according to the performance indicators. The initialization parameters include iteration period, sparsity and learning rate.

[0186] Specifically, after the above neural network model is subjected to sparse processing, the performance of the neural network model is tested, and the initialization parameters are adjusted according to the test results corresponding to the performance test, and the initialization parameters are adjusted; and the neural network model is subjected to sparse processing again according to the adjusted initialization parameters, and then the performance of the sparse neural network model is tested again until the performance test meets the preset test standard. For example, the test value corresponding to the performance test result.

[0187] Furthermore, it can be known from the above embodiments that the performance index includes multiple test indexes, such as accuracy, precision, retrieval rate, etc. Since the importance of each performance index is different, a corresponding weight can be set for each performance index, and the comprehensive index of the neural network model can be determined according to the value of the performance index and the corresponding weight, and the parameters are initialized according to the comprehensive index of the current neural network model. In the above detection method, the initialization parameters corresponding to the loop iteration according to the performance index update include:

[0188] Get the weight corresponding to the test indicator;

[0189] Determine the current comprehensive index based on the weight and the value of the test index;

[0190] Update the initialization parameters corresponding to the loop iteration based on the comprehensive indicators.

[0191] Specifically, the current comprehensive index is determined by the weight and the corresponding test index value, and the initialization parameters are adjusted according to the relationship between the current comprehensive index and the preset threshold. Through continuous adjustment, the sparse neural network model can meet the preset requirements.

[0192] See also Figure 4 , Figure 4 A schematic diagram of a method for detecting performance of a neural network chip provided in an embodiment of the present application, S105 includes:

[0193] S401. Determine initialization parameters.

[0194] Set initial values ​​for sparsity (k), learning rate (α), and epochs (C). These initial values ​​can be based on previous experience or default values.

[0195] S402. Training and evaluating the sparse neural network model.

[0196] The sparse neural network model is a neural network model loaded with the above-mentioned sparse weights.

[0197] The training model is set up using the initialization parameters, and the values ​​of various test indicators are calculated based on the training set and the validation set, including accuracy (A), recall (R), precision (P), F1 value (F1), DRAM access times (DRAM), inference speed (S), model size (M), and energy efficiency ratio (E).

[0198] S403. Calculate comprehensive indicators.

[0199] According to the importance and weight of the test indicators, the value of the comprehensive index is calculated. We use the weighted summation method and can calculate the comprehensive index (SI) using the following formula:

[0200] SI=w1*A+w2*R+w3*P+w4*F1+w5*DRAM+w6*S+w7*M+w8*E;

[0201] Among them, w1 to w8 are the weights corresponding to accuracy (A), recall (R), precision (P), F1 value (F1), DRAM access times (DRAM), inference speed (S), model size (M), and energy efficiency ratio (E). The sum of w1 to w8 is 1.

[0202] Calculate the expected comprehensive index target_SI based on the expected index:

[0203] target_SI=w1*A_ref+w2*R_ref+w3*P_ref+w4*F1_ref+w5*DRAM_ref

[0204] +w6*S_ref+w7*M_ref+w8*E_ref;

[0205] Among them, ref represents the expected value of the indicator.

[0206] S404. Analyze the test indicators.

[0207] Analyze the relationship between the values ​​of the comprehensive indicators and the values ​​of each indicator. Check whether the values ​​of the comprehensive indicators and the values ​​of each indicator meet the requirements, and determine which indicators need to be improved and which indicators have reached the expected level. If the performance of A, R, P, F1 is poor, you can consider adjusting the learning rate α and the iteration cycle C. The poor performance of DRAM, S, E may mean that the model has problems in storage resources, computing efficiency, or energy utilization by adjusting the sparsity k.

[0208] S405: Update initialization parameters.

[0209] When adjusting the initialization parameters based on the comprehensive indicator analysis results, the following strategy can be used to update the values ​​of sparsity k, learning rate α, and cycle C:

[0210] k=k+δ*(target_SI-current_SI);

[0211] α=α*(target_SI / current_SI)^β;

[0212] C=C+γ*(target_SI-current_SI);

[0213] Among them, target_SI is the expected comprehensive index value, and current_SI is the current comprehensive index value. Adjusting sparsity δ controls the adjustment step size, which can be adjusted according to needs. β is the scaled sparsity of the learning rate, which can be adjusted according to needs. Adjusting sparsity γ controls the adjustment step size, which can be adjusted according to needs. That is, according to the determined adjustment strategy, the values ​​of sparsity k, learning rate α and cycle C are updated.

[0214] Repeat the above steps to implement multiple updates of the initialization parameters until the values ​​of the comprehensive indicators and various test indicators that meet the requirements are reached.

[0215] See also Figure 5 , is a schematic diagram of the structure of a detection device provided in an embodiment of the present application. As shown in the figure, the detection device 50 provided in this embodiment may include: a processor 540, a memory 541, and a computer program 542 stored in the memory 541 and executable on the processor 540, such as a program corresponding to the detection method. When the processor 540 executes the computer program 542, the steps in the above-mentioned detection method embodiment based on the performance of the neural network chip are implemented, such as Figure 1 Steps shown.

[0216] Exemplarily, the computer program 542 may be divided into one or more modules / units, one or more modules / units are stored in the memory 541, and are executed by the processor 540 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 542 in the detection device 50.

[0217] Those skilled in the art will understand that Figure 5 The above is only an example of the detection device 50 and does not constitute a limitation of the detection device 50 , and the detection device 50 may include more or less components than those shown in the figure, or some components may be combined, or different components may be included.

[0218] The processor 540 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0219] The memory 541 may be an internal storage unit of the detection device 50, such as a hard disk or memory of the detection device 50. The memory 541 may also be an external storage device of the detection device 50, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card or a flash card (flash card) equipped on an electronic device. Further, the memory 541 may also include both an internal storage unit of the detection device 50 and an external storage device.

[0220] The memory 541 is used to store computer programs and other programs and data required by the electronic device. The memory 541 can also be used to temporarily store data that has been output or is to be output.

[0221] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure of the power supply control device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0222] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0223] An embodiment of the present application provides a computer program product. When the computer program product is run on a detection device, the detection device implements the steps in the above-mentioned various method embodiments.

[0224] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0225] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0226] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0227] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting performance of a neural network chip, characterized in that: The detection method comprises: Get the weight matrix corresponding to the neural network model; The following objective function is constructed based on the weight matrix: Where W is the weight matrix, L is the cross entropy loss function, ⊙ is the Hadamard product, M * (S) is the binary mask after the importance score is approximately binarized, ‖·‖0 is the norm of the vector L0, which is used to represent the number of non-zero elements in the vector; k is the network sparsity, S is the importance score vector, M is a preset binary mask, S * is the final importance score; W is a×b×c matrix, then p=a×b×c; a, b, c are the values ​​of different dimensions in the three-dimensional matrix respectively; Performing sparse processing on the weight matrix according to the objective function to obtain sparse weights; Loading the sparse weights into the neural network model, and performing image detection based on the neural network model; The performing image detection according to the neural network model includes: Input the test data into the weight pruned neural network model for feature extraction; Classify the extracted features, and the classification results include eyes and mouth; Determining the driver's facial features according to the classification result, the facial features including whether eyes are open or closed, and whether the mouth is open or closed; determining a fatigue state of the driver based on the facial features; The method further comprises: After the sparse weights are loaded into the neural network model, a performance test is performed on the neural network model to determine a performance indicator of the neural detection model, wherein the performance indicator includes at least one of an accuracy rate and a recall rate, wherein the accuracy rate is a correct prediction ratio when predicting whether the driver is in a fatigue state, and the recall rate is a correct prediction ratio of the driver in a fatigue state; The initialization parameters corresponding to the loop iteration are updated according to the performance indicators until the performance test of the neural detection model meets the preset test standard, and the initialization parameters include iteration period, sparsity and learning rate.

2. The detection method according to claim 1, characterized in that: The sparse processing of the weight matrix according to the objective function to obtain the sparse weights includes: The following transition function is determined according to the objective function: Among them, m i represents the i-th element in the binary mask vector M, s i is the i-th element in S, is the kth largest element in S; The weight matrix is ​​subjected to sparse processing according to the transition function to obtain sparse weights.

3. The detection method according to claim 2, characterized in that: The performing sparse processing on the weight matrix according to the transition function to obtain the sparse weights comprises: Determine the initial importance score vector; Determining an initialized binary mask according to the initialized importance score vector and the transition function; Determining initialized sparse weights according to the initialized binary mask and the weight matrix; Determining the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask according to the initialized sparsification weight; Updating the initialized importance score vector according to the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask; Then, a preset number of loop iterations of the binary mask and the importance score vector are performed to determine a sparse binary mask, and the sparse weight is determined according to the sparse binary mask.

4. The detection method according to claim 3, characterized in that: The gradient of the objective function with respect to the importance score is The gradient of the cross entropy loss function with respect to the binary mask is The updating of the initialized importance score vector according to the gradient of the objective function with respect to the importance score and the gradient of the cross entropy loss function with respect to the binary mask comprises: If the gradient of the objective function with respect to the importance score is equal to the gradient of the cross entropy loss function with respect to the binary mask, the initialized importance score vector is updated according to the following formula: Among them, t≥C, C is the iteration period, S t is the current importance score vector, S t+1 is the updated importance score vector, and α is the learning rate.

5. The detection method according to claim 3, characterized in that: The performance index includes a plurality of test indexes; The updating of the initialization parameters corresponding to the loop iteration according to the performance indicator comprises: Obtaining the weight corresponding to the test indicator; Determining the current comprehensive index according to the weight and the value of the test index; The initialization parameters corresponding to the loop iteration are updated according to the current comprehensive index.

6. The detection method according to claim 5, characterized in that: The updating of the initialization parameters corresponding to the loop iteration according to the current comprehensive index comprises: k=k+δ*(target_SI-current_SI); α=α*(target_SI / current_SI)^β; C=C+γ*(target_SI-current_SI); Among them, C is the iteration period, k is the sparsity, α is the learning rate, target_SI is the expected value of the comprehensive index, current_SI is the current value of the comprehensive index, δ is the adjustment sparsity of the sparsity, β is the scaling sparsity of the learning rate, and γ is the adjustment sparsity of the iteration period.

7. The detection method according to claim 1, characterized in that: The neural network model is a multi-task convolutional neural network model.

8. A detection device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the detection method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the detection method according to any one of claims 1 to 7 are implemented.

Images