Method for executing user-defined operator by mining embedded equipment, medium and product

Abstract

The invention discloses a method for executing a user-defined operator by mining embedded equipment, a medium and a product, and relates to the technical field of user-defined operators, and the method comprises the steps: obtaining a coal mine image sample weight through employing an initial coal mine image sample; according to the initial coal mine image sample, utilizing a smote algorithm to obtain a coal mine abnormal image data set; setting a decision threshold according to the coal mine abnormal image data set; according to the decision threshold and the coal mine image sample weight, utilizing the coal mine abnormal image data set to obtain a convolutional neural network model; and according to the convolutional neural network model, obtaining a result of executing the user-defined operator by the mining embedded device. According to the method, the weight is adjusted and the self-defined loss function is designed, so that the performance of the model on an unbalanced data set is improved, a small number of abnormal data samples can be better identified, and the identification precision of abnormal data in embedded equipment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of custom operators, and in particular to a method, medium and product for executing a custom operator on a mining embedded device. Background Art

[0002] In the process of embedded development, models trained on PCs face many problems when running on embedded devices. For example, in the field of coal mining, the normal state accounts for more than 90% of the equipment operation, and there is very little abnormal data, so algorithms are needed to accurately judge abnormalities so that they can be recognized and notified to the platform to alarm when abnormal events occur. However, traditional methods are difficult to meet the needs of efficiently identifying abnormalities on unbalanced data sets.

[0003] Therefore, there is an urgent need for a method, medium and product for executing custom operators on mining embedded devices to solve the problem of few abnormal state samples and high recognition accuracy requirements. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method, medium and product for executing custom operators on a mining embedded device. By designing a custom loss function and adjusting weights, the performance of the model on an unbalanced data set is improved, and a small number of abnormal data samples can be better identified, thereby improving the accuracy of anomaly identification during equipment operation.

[0005] On the one hand, to achieve the above-mentioned purpose, the present invention provides a method for executing a custom operator on a mining embedded device, comprising the following steps:

[0006] Using the initial coal mine image samples, obtaining coal mine image sample weights;

[0007] Acquire a coal mine abnormal image data set using a smote algorithm according to the initial coal mine image sample;

[0008] Setting a decision threshold according to the coal mine abnormal image data set;

[0009] Acquire a convolutional neural network model using the coal mine abnormal image dataset according to the decision threshold and the coal mine image sample weight;

[0010] The result of executing the custom operator on the mining embedded device is obtained according to the convolutional neural network model.

[0011] Optionally, obtaining the coal mine image sample weight using the initial image sample includes:

[0012] Collecting initial coal mine normal image data and initial coal mine abnormal image data as the initial coal mine image samples;

[0013] According to the initial coal mine image sample, an initial coal mine normal image data weight and an initial coal mine abnormal image data weight are obtained as coal mine image sample weights;

[0014] Among them, the weight of the initial coal mine abnormal image data is greater than the weight of the initial coal mine normal image data.

[0015] Optionally, obtaining a coal mine abnormal image dataset by using a smote algorithm according to the initial coal mine image sample includes:

[0016] Using the smote algorithm, the initial coal mine image samples are sequentially subjected to image preprocessing, feature extraction and neighboring sample selection to obtain coal mine abnormal image data as new samples;

[0017] Image reconstruction and data set expansion are performed in sequence according to the new samples to obtain the coal mine abnormal image data set.

[0018] Optionally, acquiring a convolutional neural network model using the coal mine abnormal image dataset according to the decision threshold and the coal mine image sample weight includes:

[0019] Acquire a target coal mine abnormal image dataset using the decision threshold and the coal mine abnormal image dataset;

[0020] Based on the target coal mine abnormal image data set, a host computer neural network is used to sequentially load and train data to obtain coal mine abnormal image training data;

[0021] Select the initial convolutional neural network model based on the embedded equipment used in mining;

[0022] Obtaining a custom fork entropy loss function according to the coal mine image sample weights;

[0023] Based on the coal mine abnormal image training data, the convolutional neural network initial model is trained using the custom fork entropy loss function to obtain the convolutional neural network model.

[0024] Optionally, using the decision threshold and the coal mine abnormal image dataset to obtain a target coal mine abnormal image dataset includes:

[0025] It is determined whether the coal mine abnormal image dataset fully meets the initial decision threshold. If so, the coal mine abnormal image dataset is obtained as the target coal mine abnormal image dataset. Otherwise, the coal mine abnormal image dataset is used for modification.

[0026] Optionally, the custom fork entropy loss function is:

[0027]

[0028] Among them, L is the custom cross entropy loss function, N is the number of samples, and w positive is the initial normal image data weight, y i Indicates the true label of the i-th sample, with a value of 0-1. represents the model's prediction result for the i-th sample, w negative is the initial abnormal image data weight.

[0029] Optionally, training the initial model of the convolutional neural network using the custom fork entropy loss function based on the coal mine abnormal image training data to obtain the convolutional neural network model includes:

[0030] Using yolo v5 to call the custom fork entropy loss function;

[0031] The convolutional neural network model is obtained by training the initial model of the convolutional neural network using the coal mine abnormal image training data according to the custom cross entropy loss function.

[0032] Optionally, obtaining the result of executing a custom operator on a mining embedded device according to the convolutional neural network model includes:

[0033] Quantizing the convolutional neural network model and porting it to an embedded device;

[0034] The custom operator is executed by using the embedded device to obtain the result of executing the custom operator by the embedded device.

[0035] On the other hand, to achieve the above-mentioned purpose, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the above-mentioned method when executed by a processor.

[0036] The present invention also provides a computer program product, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0037] Compared with the closest prior art, the present invention has the following beneficial effects:

[0038] The present invention improves the model performance on an unbalanced data set, makes the model pay more attention to abnormal data, and can have better recognition ability for a small amount of abnormal data samples on resource-constrained embedded devices; uses the smote algorithm to supplement abnormal data, solves the problem of insufficient abnormal data, and improves the accuracy of the model; and improves the performance of the model on an unbalanced data set by adjusting weights and designing a custom loss function, can better recognize a small amount of abnormal data samples, and improves the recognition accuracy of abnormal data in embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 The present invention is a flowchart of a method for executing a custom operator on a mining embedded device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention and are not intended to limit the present invention.

[0043] like Figure 1 As shown, the present invention provides a method for executing a custom operator on a mining embedded device, comprising the following steps:

[0044] S1. Obtaining coal mine image sample weights using initial coal mine image samples;

[0045] S1-1. Collect initial coal mine normal image data and initial coal mine abnormal image data as initial coal mine image samples, specifically:

[0046] First, prepare the samples. The samples here refer to normal and abnormal image data, namely, initial coal mine normal image data and initial coal mine abnormal image data. Among them, initial coal mine abnormal image data refers to the image data obtained under abnormal conditions such as foreign matter in the belt, deviation of the belt conveyor, and work surface failure. The image data collected during the safe production process are normal image data, namely, initial coal mine normal image data. Only when an abnormality occurs can there be abnormal image data. In addition, the amount of abnormal data in the sample accounts for less than one percent of all samples;

[0047] S1-2. Balance the loss function by adjusting the weights of the initial coal mine normal image data and the initial coal mine abnormal image data, and obtain the weight of the initial coal mine normal image data and the weight of the initial coal mine abnormal image data according to the initial coal mine image sample as the weight of the initial coal mine image sample, wherein this embodiment gives a larger weight to the initial coal mine abnormal image data to emphasize its importance in the loss function.

[0048] S2. Based on the initial coal mine image samples, the abnormal image data set of the smote algorithm is used. That is, the smote algorithm is used to train with artificially synthesized abnormal data to fill the problem of insufficient abnormal data and ensure the validity of the model data. The specific process includes data preparation, image preprocessing, feature extraction, neighboring sample selection, generation of new samples, image reconstruction and data set expansion. Finally, the model is trained with the balanced data set to improve its ability to recognize abnormal situations and ensure the safety of coal mine production. Specifically:

[0049] The smote algorithm is used to perform image preprocessing, feature extraction and neighboring sample selection on the initial coal mine image samples in sequence to obtain abnormal coal mine image data as new samples;

[0050] According to the new samples, image reconstruction and data set expansion are performed in sequence to obtain the coal mine abnormal image dataset.

[0051] S3. Set a decision threshold based on the coal mine abnormal image data set. By modifying the decision threshold, the model tends to abnormal data, thereby balancing the processing impact of normal and abnormal data. The decision threshold is set mainly to optimize the model and improve the decision-making ability of the system.

[0052] S4. Obtain a convolutional neural network model using the coal mine abnormal image dataset according to the decision threshold and the coal mine image sample weight, specifically:

[0053] S4-1. This embodiment mainly designs a custom operator. In the initial stage, the performance of the coal mine abnormal image dataset is detected by a preliminary decision threshold. If the performance does not meet expectations, that is, the coal mine abnormal image dataset does not fully meet the decision threshold, the coal mine abnormal image dataset is modified. Otherwise, the coal mine abnormal image dataset is obtained as the target coal mine abnormal image dataset;

[0054] S4-2, use the host computer neural network to load and train data samples, that is, obtain the abnormal image training data of coal mines. Currently, the YOLO v5 architecture platform is used, and the YOLO v5 platform is optimized to complete the training and integration of the model. In addition, some abnormal data need to be artificially synthesized to ensure that the number of normal and abnormal samples in the data set is consistent;

[0055] S4-3. Select the model size. Since it needs to run on the mining embedded device, a relatively small model is currently selected, that is, the initial model of the convolutional neural network, to adapt to the relatively small-scale tasks on the embedded device, that is, to obtain the initial model based on the mining embedded device;

[0056] S4-4. Design a custom loss function. This embodiment uses a custom fork entropy loss function.

[0057] In traditional target recognition, the choice of loss function depends on the specific problem and task to be solved. Common loss functions include mean square error and mean absolute error. The cross entropy loss function used in this embodiment can help avoid some gradient vanishing problems. It is mainly to solve the situation where low abnormal data cannot converge during training. The cross entropy loss function is used to give more weight to abnormal samples, which can usually be achieved by manually setting or parameterizing the category weights. This method can make the model pay more attention to minority categories (such as abnormal samples) during training, thereby improving the performance of the model on unbalanced data sets and the accuracy of anomaly detection; wherein, the custom cross entropy loss function L is:

[0058]

[0059] Where N is the number of samples, w positive is the initial normal image data weight, y i Indicates the true label of the i-th sample, with a value of 0-1. It represents the prediction result of the model for the i-th sample, usually after the sigmoid activation function, with a value between 0 and 1. negative is the initial abnormal image data weight.

[0060] The above formula uses the sum of two terms. The first term represents normal data, and the second term represents the loss of abnormal data samples. The loss function applies different weights to samples of different categories, so that the model pays more attention to abnormal categories during training and improves the performance of the model.

[0061] After completing the design, you need to introduce the required modules into the yolo framework and define a loss function first;

[0062] This embodiment defines a custom weighted cross entropy loss function, which is used to adjust the losses of different categories according to weights when processing classification tasks in PyTorch. The main process is: the initialization function accepts a weight list to define the loss weight of each category. During the forward propagation process, the output of the model and the true label are calculated by calculating the cross entropy loss function between the predicted output of the convolutional neural network model and the true label to calculate the weighted cross entropy loss, ensuring that the loss function takes into account the weight differences between different categories to optimize the classification performance of the model on imbalanced data sets;

[0063] S4-5. Based on the coal mine abnormal image training data, the convolutional neural network initial model is trained using a custom fork entropy loss function to obtain a convolutional neural network model;

[0064] Use the yolo framework to call your own custom loss function, and train the initial model of the convolutional neural network based on the coal mine abnormal image training data to obtain the convolutional neural network model;

[0065] Finally, the training loop is run to adapt the new loss function, the loss is calculated during training, and after the modification is completed, testing is performed to ensure that the modification did not introduce errors and that the optimization goal was achieved.

[0066] S5. Obtain the results of executing the custom operator on the mining embedded device according to the target model. Specifically:

[0067] Quantize the convolutional neural network model and port it to embedded devices;

[0068] The custom operator is executed by using the embedded device to obtain a result of executing the custom operator by the embedded device.

[0069] In the process of embedded program development, since custom operators are used in the model, and the model verification has been completed in the verification process, it meets the requirements of identifying abnormal small data volume and the requirements of the on-site target environment. At this time, quantization and transplantation are required. Therefore, the convolutional neural network model is quantified and transplanted to the embedded device, and then the embedded device is used to execute the custom operator to obtain the result of the custom operator execution on the embedded device. Since rknn provides a custom operator mechanism, it allows developers to define and execute custom operators in the reasoning stage of the rknn model, so that the parameters of the model can be modified to achieve a situation that better matches the abnormal data.

[0070] By matching model parameters with samples or model iterations, a more sophisticated model tuning and analysis process can be achieved. However, the deployment of the model needs to take into account the support of the device itself. If the tuning process can be supported, it is a detailed quantification process. The following is an example of the quantification process implemented in combination with the device:

[0071] First, you need to use RKNN-Toolkit2 to register the custom operator and export the RKNN model

[0072] Secondly, write the C code implementation of the custom operator, load it, register it and execute it through RKNNAPI.

[0073] Overall process:

[0074] 1. According to the .pt model file output by yolo v5 above, convert the model into the onnx model format, and insert it into the topology map position in onnx according to the designed WeightedCrossEntropyLoss. The user uses the API provided by the onnx package to design and export the onnx model.

[0075] 2. According to the platform requirements, you need to implement a Python class, which mainly includes two function interfaces: shape_infer() and compute(), and call rknn.reg_custom_op() to register the operator.

[0076] 3. If rknn.build() is executed successfully, you can perform inference, otherwise you need to check the shape_infer() code implementation in step 2. Then run the simulation, if the simulation result is correct, you can call the rknn.export_rknn() interface to export the RKNN model, otherwise you need to check the compute() code in step 2.

[0077] 4. Write the C code implementation of the custom operator, load and register it through RKNNAPI and execute it. According to the rknn_custom_op class in rknn_custom_op.h, write the C code implementation of the custom operator. After writing, fill in the information of the rknn_custom_op class. Register the information of the rknn_custom_op class with rknn_register_custom_ops().

[0078] Generally speaking, the added custom model needs to follow the operation of the rk platform. If the ONNX operator is not implemented in the Runtime, or the performance is poor, or the implementation is incorrect, the user can add a custom operator by replacing the ONNX operator supported by RKNN. The custom operator class only needs to set the operator type name op_type.

[0079] The embodiments of the present application also provide a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the steps of the various method embodiments of the present application.

[0080] The embodiments of the present application also provide a computer program product, including a computer program, which implements the steps of the method embodiments of the present application when executed by a processor.

[0081] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for executing a custom operator on a mining embedded device, characterized in that: The specific steps include: Using the initial coal mine image samples, obtaining coal mine image sample weights; Acquire a coal mine abnormal image data set using a smote algorithm according to the initial coal mine image sample; Setting a decision threshold according to the coal mine abnormal image data set; Acquire a convolutional neural network model using the coal mine abnormal image dataset according to the decision threshold and the coal mine image sample weight; The result of executing the custom operator on the mining embedded device is obtained according to the convolutional neural network model.

2. The method for executing a custom operator by a mining embedded device according to claim 1, characterized in that: Using the initial image samples to obtain the coal mine image sample weights includes: Collecting initial coal mine normal image data and initial coal mine abnormal image data as the initial coal mine image samples; According to the initial coal mine image sample, an initial coal mine normal image data weight and an initial coal mine abnormal image data weight are obtained as coal mine image sample weights; Among them, the weight of the initial coal mine abnormal image data is greater than the weight of the initial coal mine normal image data.

3. The method for executing a custom operator by a mining embedded device according to claim 1, characterized in that: Acquiring a coal mine abnormal image data set using the smote algorithm according to the initial coal mine image sample includes: Using the smote algorithm, the initial coal mine image samples are sequentially subjected to image preprocessing, feature extraction and neighboring sample selection to obtain coal mine abnormal image data as new samples; Image reconstruction and data set expansion are performed in sequence according to the new samples to obtain the coal mine abnormal image data set.

4. The method for executing a custom operator by a mining embedded device according to claim 1, characterized in that: Acquiring a convolutional neural network model using the coal mine abnormal image data set according to the decision threshold and the coal mine image sample weight includes: Acquire a target coal mine abnormal image dataset using the decision threshold and the coal mine abnormal image dataset; Based on the target coal mine abnormal image data set, a host computer neural network is used to sequentially load and train data to obtain coal mine abnormal image training data; Select the initial convolutional neural network model based on the embedded equipment used in mining; Obtaining a custom fork entropy loss function according to the coal mine image sample weights; Based on the coal mine abnormal image training data, the convolutional neural network initial model is trained using the custom fork entropy loss function to obtain the convolutional neural network model.

5. The method for executing a custom operator by a mining embedded device according to claim 4, characterized in that: Using the decision threshold and the coal mine abnormal image dataset to obtain a target coal mine abnormal image dataset includes: It is determined whether the coal mine abnormal image dataset fully meets the initial decision threshold. If so, the coal mine abnormal image dataset is obtained as the target coal mine abnormal image dataset. Otherwise, the coal mine abnormal image dataset is used for modification.

6. The method for executing a custom operator by a mining embedded device according to claim 4, characterized in that: The custom fork entropy loss function is: Among them, L is the custom cross entropy loss function, N is the number of samples, and w positive is the initial normal image data weight, y i Indicates the true label of the i-th sample, with a value of 0-1. represents the model's prediction result for the i-th sample, w negative is the initial abnormal image data weight.

7. The method for executing a custom operator by a mining embedded device according to claim 4, characterized in that: Based on the coal mine abnormal image training data, the convolutional neural network initial model is trained using the custom fork entropy loss function to obtain the convolutional neural network model, which includes: Using yolo v5 to call the custom fork entropy loss function; The convolutional neural network model is obtained by training the initial model of the convolutional neural network using the coal mine abnormal image training data according to the custom cross entropy loss function.

8. The method for executing a custom operator by a mining embedded device according to claim 1, characterized in that: Obtaining the results of executing the custom operator on the mining embedded device according to the convolutional neural network model includes: Quantizing the convolutional neural network model and porting it to an embedded device; The custom operator is executed by using the embedded device to obtain the result of executing the custom operator by the embedded device.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images