Industrial defect detection method and system based on model quantification

Abstract

The invention discloses an industrial defect detection method and system based on model quantification. The method comprises the following steps: accelerating and packaging a defect detection model by using a model quantification method to obtain a quantized defect detection model; setting the quantized defect detection model as a starting program of the industrial endoscope, and obtaining surrounding environment images or videos shot by the industrial endoscope; and calling the quantized defect detection model and combining with an image high-low resolution mapping reasoning method to carry out real-time reasoning detection on data of surrounding environment images or videos to obtain a detection result. The provided model acceleration method can be applied to a target detection model to realize reasoning acceleration, the accelerated model is deployed in industrial endoscope equipment, high-speed reasoning of a single-frame picture and a video stream can be carried out at the same time, meanwhile, the detection method is deployed in the industrial endoscope equipment, and the detection efficiency is improved. By means of the advantages of small size and the like of the industrial endoscope, defect detection tasks in narrow spaces, nonlinear industrial environments and other industrial environments are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of defect detection, and in particular relates to an industrial defect detection method and system based on model quantization. Background Art

[0002] As a hot research direction in the field of computer vision today, target detection has been continuously explored and studied by scientists for many years. The main purpose of the target detection task is to detect objects such as a car, a person, a dog, etc. in a picture or a video, and mark them with appropriate bounding boxes, and mark the confidence of the target and the final classification category around them. The above describes the general process of target detection tasks in the traditional sense. In different application scenarios and requirements, the output method of the final task may be quite different. Target detection is also the most essential technology for image understanding, and it is the basis for solving some other advanced visual recognition technologies, such as scene understanding, image segmentation, action recognition, etc. The application scenarios of target detection technology are very wide, including drone reconnaissance and positioning, face detection, garbage detection, etc., and there are many business processes that have been implemented in parts defect detection in industrial manufacturing and community property security.

[0003] Defect detection, as a typical application of computer vision technology in industrial production scenarios, has been developed for a long time. From the initial use of traditional image processing technology to simple machine learning technology, and then due to the development of deep learning technology and the widespread application of target detection technology, a defect detection process based on convolutional neural networks was generated. The purpose of defect detection technology is to locate and classify the positions of defective objects in the images and videos of sampled products captured in an industrial environment, and to feed back the results to the quality inspection department in combination with other software and hardware systems, so as to assist the department in completing the defect identification of the produced products. This not only helps workers quickly locate product defects and make timely rectifications, but also enables manufacturers to improve the quality of their products to a certain extent, thereby continuously improving their own product competitiveness and providing users with better services and quality assurance.

[0004] In recent years, in the process of traditional industrial production and manufacturing, due to the large production volume of workpieces within a certain period of time, enterprises simply arrange a large number of manpower to manually identify and detect cracks, pits and other defects on the surface of manufactured workpieces. On the one hand, it consumes a lot of manpower and material resources. On the other hand, the high-intensity use of the eyes by employees during the above inspection process will cause some smaller defects to be undetectable. This situation can be said to be a key factor restricting factories from improving the quality of production workpieces. At present, with the development of artificial intelligence technology, the development of intelligent manufacturing industry is vigorously promoted at home and abroad, and the application of artificial intelligence technology is becoming more and more extensive. The detection technology of using machine vision detection to accurately locate and identify surface defects of complex workpieces has gradually matured.

[0005] The progress and development of defect detection technology will eventually fall on the actual deployment and application of the model. Therefore, the research on the detection model deployment system is particularly important. However, the existing defect detection systems mostly adopt the following solution: collect images on the terminal device, and then upload them to the back-end server through wireless or wired network. The back-end server uses the detection model to process and detect the received images, and then sends the detection result information to the terminal device for display. Although the above process realizes the decoupling of the front-end image acquisition and the back-end detection processing process, it still cannot meet the scenarios with high requirements for the real-time detection results. In addition, many existing defect detection systems can often only infer single-frame images, but not video streams. Even if they can infer, due to the low FPS (frames per second), large ghosting will be generated, which will affect normal use.

[0006] In addition to the typical defect detection system introduced above, as the industrial market's requirements for defect detection systems have gradually shifted from the original accuracy-oriented to the dual-oriented accuracy and speed in recent years, more and more defect detection systems that take into account both accuracy and speed have emerged and been applied. However, these systems often use graphics cards to accelerate inference on the model inference side, which means high system deployment costs and subsequent maintenance costs. This is a considerable expense for many companies. At the same time, due to the use of graphics cards to accelerate inference, the size of the graphics card must be taken into account when designing the device size. However, this will cause large devices to be unable to be used in many industrial scenarios, such as small and confined spaces, non-linear areas, etc., causing a lot of inconvenience.

[0007] In summary, existing defect detection equipment either only considers the accuracy requirements but pays less attention to the detection speed; or, although the accuracy and speed requirements of the detection equipment are comprehensively considered, the use scenarios will be subject to many restrictions due to the large design size of the equipment. Summary of the invention

[0008] The purpose of the present invention is to overcome the problem that existing defect detection equipment is limited in scene usage while meeting the accuracy and speed requirements, and proposes an industrial defect detection method and system based on model quantization.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme: An industrial defect detection method based on model quantification comprises the following steps: The defect detection model is accelerated using the model quantization method and packaged to obtain a quantized defect detection model; The quantified defect detection model is set as the startup program of the industrial endoscope to obtain the surrounding environment image or video taken by the industrial endoscope; The quantized defect detection model is called in combination with the image high- and low-resolution mapping reasoning method to perform real-time reasoning and detection on the surrounding environment image or video data to obtain the detection results.

[0010] Furthermore, the method of accelerating the defect detection model using a model quantization method and packaging the defect detection model to obtain a quantized defect detection model includes the following steps: Perform weight conversion on the defect detection model to obtain an intermediate model Quantifying the intermediate form model to obtain a quantized model; The quantized model is encapsulated to obtain a quantized defect detection model.

[0011] Furthermore, the step of converting the weight of the defect detection model to obtain the intermediate form model specifically includes: converting the weight of the defect detection model to the intermediate form weight to obtain the intermediate form model; The weight of the defect detection model is a pth weight, the intermediate form weight is an onnx weight, and the intermediate form model is an onnx intermediate form model; The quantization is int8 quantization, and the quantized model is an onnx quantization model; The encapsulation uses C++ language and combines with OpenCV library, and defines the externally exposed function access interface during the encapsulation process.

[0012] Furthermore, the int8 quantization comprises the following steps: After determining the quantization range and finding the minimum and maximum values ​​of the weights to be quantized, use the scaling factor calculation formula and the zero point calculation formula to calculate the scaling factor and the zero point, and use the scaling factor and the zero point to convert the original floating-point weights of the onnx model into INT8 integers; The scaling factor calculation formula is as follows:

[0013] in, scale is the scaling factor, is the minimum value of the weight to be quantized, is the maximum value of the weight to be quantized, and the range of int8 quantization value is [-128,127]; The zero point calculation formula is as follows:

[0014] in, is zero point, is the maximum value of the weight to be quantized, scale is the scaling factor, the value range of int8 quantization is [-128,127]; The conversion of the original floating point weights of the onnx model into INT8 integers using the scaling factor and zero point is performed using the following formula:

[0015] in, The model weights are quantized to int8. is the weight of the intermediate form model of the onnx model, round () is used to round a number to a specified number of decimal places , forward Express Each network coefficient w Perform a loop operation. scale is the scaling factor, It is zero point.

[0016] Furthermore, the weights of the defect detection model are converted into intermediate form weights to obtain the intermediate form model: Use the PTH to ONNX tool for PTH weights, set the model path and model input size, and run the conversion script file in the terminal to obtain the converted ONNX weight file in the target path; The quantized model obtained by quantizing the intermediate form model is specifically: Use the onnx perception training quantization tool to perform quantization perception training on the onnx intermediate form model. After setting the object and quantization type of the onnx intermediate form model, run the quantization script file to obtain the onnx quantization model in the output path. The quantized defect detection model obtained by encapsulating the quantized model is specifically: Use the OpenCV library and the onnx SDK to encapsulate the weights of the onnx quantization model, use the ONNX Runtime SDK and pre-trained deep learning models to perform defect detection on surrounding environment images or videos, and visualize the defect detection results.

[0017] Furthermore, the defect detection of the surrounding environment image or video includes: creating a structure and an inference function of a defect detection model, performing defect detection on a single-frame image or performing defect detection on multiple-frame videos; the information of the structure of the defect detection model includes a model confidence threshold and an onnx model inference running session object; the inference function includes a resize_image function, a normalize function, an nms function, and a detect function; The defect detection on a single frame image comprises the following steps: Initialize the original single-frame image to obtain the initialized single-frame image. The initialization includes using the resize_image function to adjust the input size of the original single-frame image and then performing normalization operation through the normalize function. Call the CreateCpu function of the onnx SDK to allocate CPU cores for onnx model reasoning, convert the data stream of the initialized single-frame image to the input tensor format supported by the onnx SDK, and call the Run method in the onnx SDK to perform reasoning and obtain the output result. Use the nms function to perform non-maximum suppression on the output result, and draw the detected bounding box on the original single-frame image. The defect detection of multiple frames of video includes the following steps: The Mat matrix conversion tool of the OpenCV library is used to convert the YUV format of the video data output by the industrial software camera of the industrial endoscope into the cv::Mat format, and the detect function is called at the same time.

[0018] Furthermore, the image high-low resolution mapping reasoning method comprises the following steps: Before feeding the surrounding environment image or video into the quantized defect detection model, the length and width of the surrounding environment image or video are proportionally reduced to 1 / k of the original; After the quantized defect detection model outputs the result, the coordinates of the upper left corner of the bounding box (x1, y1) and the lower right corner of the bounding box (x2, y2) are increased to k times of the original.

[0019] An industrial defect detection system based on model quantization includes a packaged quantization detection module, which is used to accelerate the defect detection model using a model quantization method and encapsulate it to obtain a quantized defect detection model; An image or video acquisition module is used to set the quantized defect detection model as a startup program of the industrial endoscope and acquire an image or video of the surrounding environment taken by the industrial endoscope; The real-time reasoning detection module is used to call the quantized defect detection model and combine the image high- and low-resolution mapping reasoning method to perform real-time reasoning detection on the surrounding environment image or video data to obtain the detection results.

[0020] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the industrial defect detection method based on model quantization when executing the computer program.

[0021] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the industrial defect detection method based on model quantization is implemented.

[0022] Compared with the prior art, the present invention has the following beneficial technical effects: The present invention provides an industrial defect detection method based on model quantization, which can apply the proposed model acceleration method to the target detection model to realize reasoning acceleration. The target detection model can be based on the yolov5s target detection model built on PyTorch, and the accelerated model is deployed in the industrial endoscope equipment, which can simultaneously perform high-speed reasoning of single-frame images and video streams. At the same time, the detection method is deployed in the industrial endoscope equipment, and the advantages of the small size of the industrial endoscope and the software camera carried at the end can be used to realize the defect detection tasks in special industrial environments such as small spaces and non-linear shapes.

[0023] Furthermore, the proposed model quantization method is used to quantize the deep learning model. It is only necessary to put the quantized model into the corresponding file directory, replace the original model, and restart the defect detection equipment to complete the model replacement process. When the target detection model is deployed in the industrial endoscope equipment, the model can be quickly replaced, so that the defect detection system has the ability to solve defect detection problems in different industrial scenarios in a targeted manner.

[0024] Furthermore, the industrial defect detection method based on model quantization provided by the present invention specifically solves the problem of poor real-time performance of the system detection caused by the fact that the reasoning and imaging of the existing industrial defect detection system are not on the same end. The defect detection model is quantized by the proposed model quantization method and the combination of single-frame image and multi-frame video stream reasoning is realized. At the same time, the proposed image high- and low-resolution mapping reasoning method is used during reasoning, which further accelerates the reasoning speed of the defect detection system while maintaining the accuracy without significant changes.

[0025] Furthermore, the industrial defect detection method based on model quantization proposed in the present invention is also integrated into the industrial endoscope, which can realize the defect detection task in a small space and complex industrial scene with the help of the industrial software camera loaded on the industrial endoscope. The proposed industrial defect detection method based on model quantization exceeds the method deployed in the traditional way in terms of speed, and is basically the same as the method deployed in the traditional way in terms of accuracy. The proposed industrial defect detection method and system based on model quantization make up for the shortcomings of the current existing technical solutions for industrial defect detection to a certain extent, and provide a reliable technical solution for defect detection tasks in complex industrial scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are for explanation purposes only and are not intended to limit the scope of the present invention in any way. In addition, the shapes and proportional dimensions of the components in the drawings are only for illustration purposes to help understand the present invention, and are not intended to specifically limit the shapes and proportional dimensions of the components of the present invention. In the drawings: Figure 1 The present invention is a flow chart of an industrial defect detection method based on model quantization.

[0027] Figure 2 This is a structural diagram of an industrial defect detection system based on model quantization of the present invention.

[0028] Figure 3 This is an electronic device diagram of an industrial defect detection method based on model quantization according to the present invention.

[0029] Figure 4 This is an overall flow chart of an industrial defect detection method based on model quantization in Example 6 of the present invention.

[0030] Figure 5 This is a schematic diagram of the industrial endoscope used in Example 6 of the present invention.

[0031] Figure 6 This is a schematic diagram of model quantization accelerated reasoning for an industrial defect detection method based on model quantization in Example 6 of the present invention.

[0032] Figure 7 This is a timing diagram of model quantization accelerated reasoning for an industrial defect detection method based on model quantization in Example 6 of the present invention.

[0033] Figure 8 Schematic diagram of an image high- and low-resolution mapping reasoning method used in model quantization accelerated reasoning of an industrial defect detection method based on model quantization in Example 6 of the present invention.

[0034] Fig. 9This is a schematic diagram of an industrial defect detection method based on model quantization in Example 6 of the present invention performing defect detection in a complex industrial scenario in a narrow space.

[0035] Fig.10 Schematic diagram of a metal defect data set used to verify the proposed defect detection method in Example 6 of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] Embodiment 1 See also Figure 1 , an industrial defect detection method based on model quantification, comprising the following steps: The defect detection model is accelerated using the model quantization method and packaged to obtain a quantized defect detection model; The quantified defect detection model is set as the startup program of the industrial endoscope to obtain the surrounding environment image or video taken by the industrial endoscope; The quantized defect detection model is called in combination with the image high- and low-resolution mapping reasoning method to perform real-time reasoning and detection on the surrounding environment image or video data to obtain the detection results.

[0038] This embodiment can significantly reduce the size and computational complexity of the model by quantizing the trained defect detection model, thereby improving the reasoning speed. This means that under the same hardware conditions, images or video streams can be processed faster to achieve real-time or near real-time defect detection; the quantized defect detection model is encapsulated as an executable file and set as the default startup program of the industrial endoscope host, making the entire detection system more modular and easy to deploy; combined with the image high- and low-resolution mapping reasoning method, it can improve the accuracy of detection while ensuring the detection speed and utilizing the detail information in the high-resolution image.

[0039] Optionally, the industrial endoscope includes an industrial software camera and an industrial endoscope host. The image or video of the surrounding environment is captured by the industrial software camera, and the detection results are displayed in real time on the display screen of the industrial endoscope host. The industrial endoscope host is equipped with an operating system, and the industrial software camera uses a fiber beam for image transmission and light guidance. The quantified defect detection model is set as the startup program of the industrial endoscope, specifically: the quantified defect detection model is packaged as an executable file; the executable file is stored in the file directory of the industrial endoscope host, and the default startup program of the industrial endoscope host is modified to the quantified defect detection model. The detection results are displayed in real time on the display screen of the industrial endoscope host, so that the operator can immediately understand the detection situation and respond in time, which helps to improve the quality control level of the production line and reduce the production of defective products.

[0040] The defect detection model is accelerated by the model quantization method and packaged to obtain a quantized defect detection model, which includes the following steps: Perform weight conversion on the defect detection model to obtain an intermediate model Quantifying the intermediate form model to obtain a quantized model; The quantized model is encapsulated to obtain a quantized defect detection model.

[0041] The weight conversion of the defect detection model to obtain the intermediate form model is specifically as follows: the weight of the defect detection model is converted into the intermediate form weight to obtain the intermediate form model; The weight of the defect detection model is the pth weight, the intermediate form weight is the onnx weight, and the intermediate form model is the onnx intermediate form model; The quantization is int8 quantization, and the quantized model is onnx quantization model; The encapsulation uses C++ language and combines with OpenCV library, and defines the externally exposed function access interface during the encapsulation process.

[0042] Int8 quantization includes the following steps: After determining the quantization range and finding the minimum and maximum values ​​of the weights to be quantized, use the scaling factor calculation formula and the zero point calculation formula to calculate the scaling factor and the zero point, and use the scaling factor and the zero point to convert the original floating-point weights of the onnx model into INT8 integers; The scaling factor calculation formula is as follows:

[0043] in, scale is the scaling factor, is the minimum value of the weight to be quantized, is the maximum value of the weight to be quantized, and the range of int8 quantization value is [-128,127]; The zero point calculation formula is as follows:

[0044] in, is zero point, is the maximum value of the weight to be quantized, scale is the scaling factor, the value range of int8 quantization is [-128,127]; The original floating point weights of the onnx model are converted to INT8 integers using the scaling factor and zero point using the following formula:

[0045] in, The model weights are quantized to int8. is the weight of the intermediate form model of the onnx model, round () is used to round a number to a specified number of decimal places , forward Express Each network coefficient w Perform a loop operation. scale is the scaling factor, It is zero point.

[0046] The weights of the defect detection model are converted into intermediate form weights to obtain the intermediate form model: Use the PTH to ONNX tool for PTH weights, set the model path and model input size, and run the conversion script file in the terminal to obtain the converted ONNX weight file in the target path; The quantized model obtained by quantizing the intermediate form model is as follows: Use the onnx perception training quantization tool to perform quantization perception training on the onnx intermediate form model. After setting the object and quantization type of the onnx intermediate form model, run the quantization script file to obtain the onnx quantization model in the output path. The quantized model is encapsulated to obtain the quantized defect detection model: Use the OpenCV library and the onnx SDK to encapsulate the weights of the onnx quantization model, use the ONNX Runtime SDK and pre-trained deep learning models to perform defect detection on surrounding environment images or videos, and visualize the defect detection results.

[0047] Defect detection on the surrounding environment image or video includes: creating the structure and inference function of the defect detection model, performing defect detection on a single frame image or on multiple frames of video; the information of the defect detection model structure includes the model confidence threshold and the onnx model inference running session object; the inference function includes the resize_image function, the normalize function, the nms function and the detect function; Defect detection for a single frame image includes the following steps: Initialize the original single-frame image to obtain the initialized single-frame image. The initialization includes using the resize_image function to adjust the input size of the original single-frame image and then performing normalization operation through the normalize function. Call the CreateCpu function of the onnx SDK to allocate CPU cores for onnx model reasoning, convert the data stream of the initialized single-frame image to the input tensor format supported by the onnx SDK, and call the Run method in the onnx SDK to perform reasoning and obtain the output result. Use the nms function to perform non-maximum suppression on the output result, and draw the detected bounding box on the original single-frame image. Defect detection for multiple frames of video includes the following steps: The Mat matrix conversion tool of the OpenCV library is used to convert the YUV format of the video data output by the industrial software camera of the industrial endoscope into the cv::Mat format, and the detect function is called at the same time.

[0048] The image high- and low-resolution mapping reasoning method includes the following steps: Before feeding the surrounding environment image or video into the quantized defect detection model, the length and width of the surrounding environment image or video are proportionally reduced to 1 / k of the original; After the quantized defect detection model outputs the result, the coordinates of the upper left corner of the bounding box (x1, y1) and the lower right corner of the bounding box (x2, y2) are increased to k times of the original.

[0049] Embodiment 2 See also Figure 2 , an industrial defect detection system based on model quantization, including a packaged quantization detection module, for accelerating a defect detection model using a model quantization method and encapsulating it to obtain a quantized defect detection model; An image or video acquisition module is used to set the quantized defect detection model as a startup program of the industrial endoscope and acquire an image or video of the surrounding environment taken by the industrial endoscope; The real-time reasoning detection module is used to call the quantized defect detection model and combine the image high- and low-resolution mapping reasoning method to perform real-time reasoning detection on the surrounding environment image or video data to obtain the detection results.

[0050] Embodiment 3 See also Figure 3 , an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an industrial defect detection method based on model quantization as described in Example 1 is implemented.

[0051] Embodiment 4 A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for industrial defect detection based on model quantization described in Example 1 is implemented.

[0052] Embodiment 5 In order to better illustrate the various components of the industrial defect detection method and system based on model quantization proposed in this embodiment, Figure 4 The figure is an overall flow chart of the industrial defect detection method based on model quantization. The overall process is: the trained defect detection model is accelerated by the model quantization method proposed in this embodiment, and then the quantized model is encapsulated using the C++ language and combined with the OpenCV library, and the function access interface is exposed to the outside, and then packaged into an executable file in .exe format and stored in the file directory of the industrial endoscope host. At the same time, the default startup program of the industrial endoscope host is modified to the .exe executable file just packaged. After the industrial endoscope is started, the defect detection system program is executed by default. The program will actively activate and open the industrial software camera to shoot the surrounding environment. Then, the captured image or video stream is transmitted to the defect detection system, and the detection system calls the quantized model to perform real-time reasoning detection on the data. Finally, the detection results will be displayed in real time on the display screen of the industrial endoscope host, giving users the most reliable and rapid feedback in a timely manner.

[0053] The industrial defect detection method based on model quantization mainly includes two parts: industrial defect detection method and system and industrial endoscope equipment for actual deployment.

[0054] Industrial endoscope equipment consists of two parts: industrial software camera and industrial endoscope host. The former is mainly used to collect images and video streams in industrial scenes, and the latter is used to integrate model reasoning detection system. Figure 5The figure shows a schematic diagram of the industrial endoscope used in this embodiment. The industrial software camera carried by it transmits images and guides light by fiber beams, and has good flexibility and convenient operation performance. The host part of the industrial endoscope is equipped with Windows 10 system, which is easy to operate and provides multiple file directories and large-capacity storage for model deployment.

[0055] Figure 6 FIG. 1 is a schematic diagram of the model quantization (accelerated reasoning) method proposed in this embodiment. The proposed model quantization method specifically consists of the following parts: Convert the model pth weight to the onnx intermediate form model, perform int8 quantization on the onnx intermediate form model, use C++ to encapsulate the quantized .onnx model (which involves adapting single-frame images or multi-frame video streams), and the proposed image high- and low-resolution mapping inference method (which can further speed up the model inference speed).

[0056] The specific steps and implementation process of each part will be explained in detail below.

[0057] (1) Convert the model PTH weights to the intermediate form of ONNX. This step is mainly to facilitate the quantization of the trained PTH model weights. Although the PTH weights can also be quantized directly, since the PTH file not only retains the model parameters and other information, but also includes other useless information such as training parameters, if the PTH weights are directly quantized, the model's inference speed cannot meet the requirements. Specifically, use the PTH to ONNX tool provided in the PyTorch framework, set the model path and model input size, and then run the py script file in the terminal. Finally, the corresponding converted ONNX weight file will be obtained under the target path.

[0058] (2) Perform int8 quantization on the onnx intermediate model. This step mainly quantizes the onnx model weights obtained in the previous step to the int8 type, thereby reducing the amount of calculation and resource usage of the model during inference. Specifically, use the quantize_qat perception training quantization tool officially launched by onnx, and run the py script file after setting the onnx model object and the quantization type to "QuantType.QInt8", and you can get the quantized onnx weights in the corresponding output path.

[0059] The following is a detailed description of the calculation formula and steps for int8 quantization of the onnx model: First, assuming that there is a model weight weights , determine the quantization range (in int8 quantization, the value range is usually [-128, 127]), find weights This step is implemented using the following formulas (1) and (2): (1) (2) Then, the scaling factor (scale) and the zero point (zero point) are calculated as shown in the following formulas (3) and (4): (3) (4) Afterwards, the scaling factor and zero point calculated above are applied to the model, as shown in the following formula (5): (5) Finally, we get the quantized model weights .

[0060] (3) Use C++ to encapsulate the quantized onnx model (which involves adapting a single-frame image or a multi-frame video stream). This step uses the OpenCV library and the onnx SDK to encapsulate the model generated in step (2). Model weights. Specifically, since single-frame or multi-frame video stream reasoning detection is required in many scenarios, adaptation is required.

[0061] For a single-frame scenario, first create a structure corresponding to the defect detection model. The structure contains information such as: model confidence threshold (which allows users to flexibly set different thresholds according to different scenarios), onnx model inference runtime session object (mainly used for onnx model inference intermediate state switching, etc.). Then, create a series of inference functions, including: resize_image, normalize, nms, and detect. Among them, the detect function implements the complete process from receiving incoming data to calling the model to the model output results. The detailed timing is as follows: Figure 7 shown.

[0062] from Figure 7 As can be seen, the image is initialized first, which requires adjusting the image input size, and then the image is normalized. Then, the CreateCpu function in the Ort::MemoryInfo class included in the onnx SDK is called to allocate available CPU cores for onnx model reasoning. After that, the data stream is converted to the input tensor format supported by the onnx SDK, and the Run method in the onnx SDK is called for reasoning and the corresponding output results are obtained. Then, the output result is used to perform the non-maximum suppression process using the created nms function, and the detected bounding box is drawn on the original image.

[0063] For multi-frame video stream scenarios, in particular, since the industrial software camera used in the industrial endoscope outputs video streams in YUV format, it is necessary to use the Mat matrix conversion tool in the OpenCV library to convert the YUV format data into cv::Mat format. Then, call the detect function described above at the same time to realize multi-frame video stream inference.

[0064] (4) Image high- and low-resolution mapping reasoning method (can further speed up model reasoning). By sending the image stream captured by the industrial software camera to the onnx quantized model before reasoning, the image length and width are reduced to 1 / k of the original, and then after the onnx model outputs the result, the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) of the bounding box are increased to k times the original. The schematic diagram of the image high- and low-resolution mapping reasoning method used is shown in Figure 8 shown.

[0065] The following is a detailed description of the process of replacing the model in the industrial defect detection system based on model quantization. When the model needs to be replaced, it is only necessary to convert the new PTH model weights using the PTH to ONNX tool provided in the PyTorch framework to obtain an ONNX model, and then use the above-mentioned model quantization acceleration method to quantize the obtained ONNX model. After that, the quantized model is used to replace the original ONNX quantized model stored in the / model directory corresponding to the industrial endoscope host. Through the above process, any model can be replaced.

[0066] After that, the exe executable program obtained by the above process of encapsulating the onnx quantization model using C++ is stored in the system startup directory of the industrial endoscope host, so that the industrial endoscope can automatically execute the industrial defect detection program when it is turned on. At the same time, with the help of the industrial software camera carried, it can realize the detection of defects in small and complex industrial scene spaces such as the inner wall of the pipeline. The schematic diagram of using the industrial defect detection method based on model quantization proposed in this embodiment to perform defect detection in a small space and complex industrial scene is shown in the figure. Fig. 9 shown.

[0067] In order to further verify that the proposed industrial defect detection method based on model quantization is highly effective, a metal surface defect test dataset containing a sufficient number of defects is used to verify the accuracy and speed of the proposed detection method.

[0068] The metal surface defect dataset contains a total of 2100 metal plate images with a size of 1024×1024, such as Fig.10As shown, each picture contains three defects of varying numbers, namely scratches, grooves and rust. The above data set is divided into 7:2:1 (training set: validation set: test set) ratios, and the model is trained using the training set. The process follows the yolov5s model description document. Then, the trained model pth weight is quantized and encapsulated using the above model quantization method, and deployed in the industrial endoscope host to obtain the final defect detection system. Afterwards, during the test, the divided metal surface defect test data set is used for testing. It is emphasized here that the industrial defect detection method and system based on model quantization proposed in this patent can achieve an inference speed of 35 frames per second (i.e., FPS is 35) in a CPU environment when the image resolution is 512×512. It far exceeds the speed of inference using pth weights in a GPU graphics card environment - 15 frames per second (i.e., FPS is 15). At the same time, the industrial defect detection method proposed in this embodiment can achieve a defect recall rate of 92% in a CPU inference environment, which is close to the recall rate of -95% when inference using pth weights in a GPU graphics card environment.

[0069] In summary, the common problems currently existing in industrial defect detection scenarios are: the detection equipment and the front-end and back-end of the reasoning process are decoupled, and real-time detection of defects cannot be achieved; even if the defect detection equipment can achieve real-time detection of defects, it often uses a graphics card GPU for reasoning acceleration, which makes the defect detection equipment larger in size and cannot achieve defect detection in small spaces and complex industrial scenarios.

[0070] The industrial defect detection method based on model quantization proposed in this embodiment accelerates the trained defect detection model through the model quantization method proposed in this embodiment, and then encapsulates the quantized model using C++ language and OpenCV library, exposes the function access interface to the outside, and then packages it into an executable file in .exe format and stores it in the file directory of the industrial endoscope host. At the same time, modify the default startup program of the industrial endoscope host to the .exe executable file that has just been packaged. After the industrial endoscope is started, the defect detection system program is executed by default. The program will actively activate and open the industrial software camera to shoot the surrounding environment. Then, the captured image or video stream is transmitted to the defect detection system, and the detection system calls the quantized model to perform real-time reasoning detection on the data. Finally, the detection results will be displayed in real time on the display screen of the industrial endoscope host, giving users the most reliable and rapid feedback in a timely manner.

[0071] The defect detection results obtained through the above process have high accuracy and high defect detection rate, and the detection speed is faster than the common model deployment method. With the help of the industrial software camera carried by the industrial endoscope, it is possible to detect defects in small spaces and complex industrial scenes. To a certain extent, it makes up for the shortcomings of the current technical solutions for industrial defect detection and provides a reliable technical solution for defect detection tasks in complex industrial scenes.

[0072] Those skilled in the art will appreciate that the embodiments of the present embodiment may be provided as methods, systems, or computer program products. Therefore, the present embodiment may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present embodiment may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0073] The present embodiment is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present embodiment. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of this embodiment rather than to limit it. Although this embodiment is described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation of this embodiment can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not deviate from the spirit and scope of this embodiment should be included in the scope of protection of the claims of this embodiment.

Claims

1. An industrial defect detection method based on model quantification, characterized in that: The following steps are involved: The defect detection model is accelerated using the model quantization method and packaged to obtain a quantized defect detection model; The quantified defect detection model is set as the startup program of the industrial endoscope to obtain the surrounding environment image or video taken by the industrial endoscope; The quantized defect detection model is called in combination with the image high- and low-resolution mapping reasoning method to perform real-time reasoning and detection on the surrounding environment image or video data to obtain the detection results.

2. The industrial defect detection method based on model quantization according to claim 1 is characterized in that: The method of accelerating the defect detection model using the model quantization method and packaging the defect detection model to obtain a quantized defect detection model includes the following steps: Perform weight conversion on the defect detection model to obtain an intermediate model Quantifying the intermediate form model to obtain a quantized model; The quantized model is encapsulated to obtain a quantized defect detection model.

3. The industrial defect detection method based on model quantization according to claim 2 is characterized in that: The step of converting the weight of the defect detection model to obtain the intermediate form model specifically includes: converting the weight of the defect detection model to the intermediate form weight to obtain the intermediate form model; The weight of the defect detection model is a pth weight, the intermediate form weight is an onnx weight, and the intermediate form model is an onnx intermediate form model; The quantization is int8 quantization, and the quantized model is an onnx quantization model; The encapsulation uses C++ language and combines with OpenCV library, and defines the externally exposed function access interface during the encapsulation process.

4. The industrial defect detection method based on model quantization according to claim 3 is characterized in that: The int8 quantization includes the following steps: After determining the quantization range and finding the minimum and maximum values ​​of the weights to be quantized, use the scaling factor calculation formula and the zero point calculation formula to calculate the scaling factor and the zero point, and use the scaling factor and the zero point to convert the original floating-point weights of the onnx model into INT8 integers; The scaling factor calculation formula is as follows: in, scale is the scaling factor, is the minimum value of the weight to be quantized, is the maximum value of the weight to be quantized, and the range of int8 quantization value is [-128,127]; The zero point calculation formula is as follows: in, is zero point, is the maximum value of the weight to be quantized, scale is the scaling factor, the value range of int8 quantization is [-128,127]; The conversion of the original floating point weights of the onnx model into INT8 integers using the scaling factor and zero point is performed using the following formula: in, The model weights are quantized to int8. is the weight of the intermediate form model of the onnx model, round () is used to round a number to a specified number of decimal places , forward Express Each network coefficient w Perform a loop operation. scale is the scaling factor, It is zero point.

5. The industrial defect detection method based on model quantization according to claim 3 is characterized in that: The weights of the defect detection model are converted into intermediate form weights to obtain the intermediate form model: Use the PTH to ONNX tool for PTH weights, set the model path and model input size, and run the conversion script file in the terminal to obtain the converted ONNX weight file in the target path; The quantized model obtained by quantizing the intermediate form model is specifically: Use the onnx perception training quantization tool to perform quantization perception training on the onnx intermediate form model. After setting the object and quantization type of the onnx intermediate form model, run the quantization script file to obtain the onnx quantization model in the output path. The quantized defect detection model obtained by encapsulating the quantized model is specifically: Use the OpenCV library and onnx SDK to encapsulate the weights of the onnx quantization model, use the ONNX Runtime SDK and pre-trained deep learning models to perform defect detection on surrounding environment images or videos, and visualize the defect detection results.

6. The industrial defect detection method based on model quantization according to claim 5, characterized in that: The defect detection of the surrounding environment image or video includes: creating a structure and an inference function of a defect detection model, performing defect detection on a single frame image or performing defect detection on multiple frames of video; the information of the structure of the defect detection model includes a model confidence threshold and an onnx model inference running session object; the inference function includes a resize_image function, a normalize function, an nms function and a detect function; The defect detection on a single frame image comprises the following steps: Initialize the original single-frame image to obtain the initialized single-frame image. The initialization includes using the resize_image function to adjust the input size of the original single-frame image and then performing normalization operation through the normalize function. Call the CreateCpu function of onnxSDK to allocate CPU cores for onnx model reasoning, convert the data stream of the initialized single-frame image to the input tensor format supported by onnx SDK, and call the Run method in onnx SDK to perform reasoning and obtain the output result. Use the nms function to perform non-maximum suppression on the output result, and draw the detected bounding box on the original single-frame image. The defect detection of multiple frames of video includes the following steps: The Mat matrix conversion tool of the OpenCV library is used to convert the YUV format of the video data output by the industrial software camera of the industrial endoscope into the cv::Mat format, and the detect function is called at the same time.

7. The industrial defect detection method based on model quantization according to claim 6 is characterized in that: The image high-low resolution mapping reasoning method comprises the following steps: Before feeding the surrounding environment image or video into the quantized defect detection model, the length and width of the surrounding environment image or video are proportionally reduced to 1 / k of the original; After the quantized defect detection model outputs the result, the coordinates of the upper left corner of the bounding box (x1, y1) and the lower right corner of the bounding box (x2, y2) are increased to k times of the original.

8. An industrial defect detection system based on model quantization, characterized in that: It includes a packaging quantization detection module, which is used to accelerate the defect detection model using a model quantization method and obtain a quantized defect detection model after packaging; An image or video acquisition module is used to set the quantized defect detection model as a startup program of the industrial endoscope and acquire an image or video of the surrounding environment taken by the industrial endoscope; The real-time reasoning detection module is used to call the quantized defect detection model and combine the image high- and low-resolution mapping reasoning method to perform real-time reasoning detection on the surrounding environment image or video data to obtain the detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a model-quantization-based industrial defect detection method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the industrial defect detection method based on model quantization as described in any one of claims 1 to 7.

Images