A method for industrial dimension measurement

Abstract

The present invention relates to an industrial dimension measurement method, comprising the following steps: collecting an image of an object to be detected; extracting multi-scale features in the image, and a prediction candidate frame corresponding to the multi-scale features; cropping the corresponding features in the prediction candidate frame, and obtaining the type and position information of each prediction candidate frame through a fully connected network according to the cropped features; inputting all cropped features into a key point detection branch, and obtaining the horizontal coordinate value information, vertical coordinate value information and visibility information of all type points in each feature; matching the type of the prediction candidate frame with the type point, and filtering out the non-existent points through network prediction visibility, and obtaining the type and coordinates of the final prediction point. The present invention can detect points in the target frame while detecting the target, and only needs to predict once, and the algorithm is fast, and can detect the size of industrial parts in real time under high-speed production.

Description

Technical Field

[0001] The invention relates to the technical field of industrial part size measurement, in particular to an industrial size measurement method. Background Art

[0002] Industrial part dimensional measurement is a critical step in the manufacturing and quality control process. Correct dimensional measurement ensures that parts meet design specifications and meet quality standards. If part dimensions are incorrect, it will lead to defects in downstream products. In some high-speed operation occasions, traditional detection methods (including measurement based on calipers, micrometers, ultrasonic measurement, X-ray detection and other tools and methods) and existing AI detection solutions are difficult to meet the needs of real-time operation.

[0003] Existing AI-based measurements mainly use the detection plus classification method, that is, the target detection model is first used to detect the position of the edge to be measured in the image, and then the detection frame cropped image is sent to the key point detection model to detect the endpoints of the line segments in the frame. If there are multiple edges to be detected in the image to be inspected, the key point detection model will be run repeatedly multiple times, which is difficult to meet the needs of real-time detection under high-speed production conditions. In addition, this solution requires two data sets and two AI models, and the operation is relatively cumbersome. Although only the key point model can be used when the position of the edge to be detected is relatively fixed, the same image must be run multiple times when detecting multiple edges, making it difficult to detect the size of industrial parts in real time under high-speed production. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an industrial dimension measurement method in view of the above shortcomings. The present invention can detect points in the target frame while detecting the target, and only needs to predict once. The algorithm is fast and can detect the size of industrial parts in real time under high-speed production.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] An industrial dimension measurement method comprises the following steps:

[0007] Collect images of objects to be inspected;

[0008] Extract multi-scale features in the image and predict candidate boxes corresponding to the multi-scale features;

[0009] Crop the corresponding features in the predicted candidate box, and obtain the type and location information of each predicted candidate box through the fully connected network based on the cropped features;

[0010] All the cropped features are input into the key point detection branch to obtain the horizontal coordinate value information, vertical coordinate value information and visibility information of all types of points in each feature;

[0011] The type of the predicted candidate box is matched with the type point, and the non-existent points are filtered out through network prediction visibility to obtain the type and coordinates of the final predicted point.

[0012] Furthermore, the method for extracting multi-scale features in an image includes: inputting the image into a feature extraction network and a feature pyramid network to obtain multi-scale features;

[0013] The method for extracting a prediction candidate box corresponding to a multi-scale feature includes: passing the multi-scale feature through an RPN network to obtain a prediction candidate box.

[0014] Furthermore, after cropping the corresponding features in the predicted candidate box, the cropped features are uniformly scaled to a size of 7×7 through ROI line 1. After the scaled features are flattened, the type and position information of the predicted candidate box are obtained through a fully connected network.

[0015] Furthermore, before all the cropped features are input into the key point detection branch, the key point detection branch is also trained. Multiple image samples are collected, and the features in the image samples are selected through real frame labels to obtain a feature training set. The feature training set is input into the key point detection branch. After the training is completed, a trained key point detection branch is obtained.

[0016] Furthermore, the key point detection branch includes the following steps:

[0017] All cropped features are uniformly scaled to 28×28 size through 28×28 ROI line 2, and all features a of size (n, 64, 28, 28) are obtained, where n is the number of features in all predicted candidate boxes;

[0018] Input all features in feature a into the ordinate classification head to obtain the ordinate value information of all types of points of size (c2,112), where c2 is the total number of types of all points and 112 is 28×4 coordinate information;

[0019] Input all features in feature a into the horizontal coordinate classification head to obtain the horizontal coordinate value information of all types of points of size (c2,112), where c2 is the total number of types of all points and 112 is 28×4 coordinate information;

[0020] All features in feature a are input into the visibility classification head to obtain the visibility information of all types of points of size (c2, 2), where c2 is the total number of types of all points and 2 is the visibility state of the point.

[0021] Furthermore, after the features in feature a are input into the ordinate classification head, the following steps are included:

[0022] After the features in feature a are convolved by Conv2D_y and activated by ReLU, the image feature y1 is obtained;

[0023] After the feature y1 passes through the Conv2D_y convolution and ReLU activation function, the image feature y2 is obtained;

[0024] After flattening feature y2, we get feature y3;

[0025] After the feature y3 passes through one-dimensional deconvolution and ReLU activation function, feature y4 is obtained;

[0026] After the feature y4 passes through the one-dimensional convolution and ReLU activation function, the vertical coordinate value information of the type point is obtained.

[0027] Furthermore, after the features in feature a are input into the abscissa classification head, the following steps are included:

[0028] After the features in feature a are convolved by Conv2D_x and activated by ReLU, image feature x1 is obtained;

[0029] After the feature x1 passes through the Conv2D_x convolution and the ReLU activation function, the image feature x2 is obtained;

[0030] After flattening feature x2, we get feature x3;

[0031] After the feature x3 passes through one-dimensional deconvolution and ReLU activation function, feature x4 is obtained;

[0032] After the feature x4 passes through the one-dimensional convolution and ReLU activation function, the horizontal coordinate value information of the type point is obtained.

[0033] Furthermore, after the features in feature a are input into the visibility classification head, the following steps are included:

[0034] The features in feature a are subjected to two Conv2D convolutions and ReLU activation functions, and then to global pooling, flattening, and full connection to obtain the visibility information of the type point.

[0035] Furthermore, the horizontal coordinate information and the vertical coordinate information of all types of points are subjected to the Soft-argmax function to obtain the coordinates of all types of points.

[0036] Furthermore, the method for matching the type of the predicted candidate box with the type point includes the following steps:

[0037] Set the matching relationship between the box type and the point type;

[0038] Sets the visibility of type points that match the box type to Visible, and points that do not match to Invisible.

[0039] After adopting the above technical solution, the present invention has the following advantages compared with the prior art:

[0040] The present invention can detect points in the target frame while detecting the target, and can realize the function of detecting characteristic type points by a specific type of detection frame;

[0041] When capturing multiple edges, the method of the present invention only needs to predict once. Compared with the method of first detecting and then capturing edges, which requires running the model multiple times, the algorithm is faster.

[0042] The present invention is described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall flow chart of the present invention;

[0044] Figure 2 It is a schematic diagram of the flow chart of the key point detection branch of the present invention;

[0045] Figure 3 It is a marked schematic diagram of the present invention;

[0046] Figure 4 This is a visualization diagram of the network prediction of the present invention. Figure 1 ;

[0047] Figure 5 This is a visualization diagram of the network prediction of the present invention. Figure 2 . DETAILED DESCRIPTION

[0048] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0049] An industrial dimension measurement method comprises the following steps:

[0050] Collect images of objects to be inspected;

[0051] Extract multi-scale features in the image and predict candidate boxes corresponding to the multi-scale features;

[0052] Crop the corresponding features in the predicted candidate box, and obtain the type and location information of each predicted candidate box through the fully connected network based on the cropped features;

[0053] All the cropped features are input into the key point detection branch to obtain the horizontal coordinate value information, vertical coordinate value information and visibility information of all types of points in each feature;

[0054] The type of the predicted candidate box is matched with the type point, and the non-existent points are filtered out through network prediction visibility to obtain the type and coordinates of the final predicted point.

[0055] Among them, the multi-scale feature is the image feature information extracted at different scales, and the cropped feature is the feature cropped to the corresponding size according to the size of the predicted candidate box itself.

[0056] Embodiment 1:

[0057] The present invention proposes a new Pose RCNN model, which is used to detect the endpoints of the line segments in the detection frame while detecting the edges. We add a key point detection branch based on Faster RCNN to detect the key points in the prediction frame, so that a model can detect the edge and obtain the precise position of the edge at the same time. By obtaining the detailed positions of the endpoints of each type of prediction frame and each type of straight line in the prediction frame on the part image, the reliable position of the contour line of the part can be obtained, and the detailed size of the part can be obtained. Our solution uses Faster RCNN to detect the straight line frame, but is not limited to Faster RCNN.

[0058] An industrial dimension measurement network model, such as Figure 1 As shown in Figure 2, the key point detection branch is added based on Faster RCNN. Figure 2 As shown in the figure, a proposa l prediction process is taken as an example.

[0059] The input image is passed through the feature lifting network and feature pyramid network (FPN) to obtain multi-scale features, and the multi-scale features are passed through the RPN (Region Proposal Networks) network to obtain the prediction candidate box (proposa l). Each prediction candidate box cuts out features from the features of the corresponding size according to its own size, and all the cut features are uniformly scaled to a size of 7×7 through ROI line 1. The 7×7 features corresponding to each prediction candidate box are flattened and connected to the fully connected network to obtain the type and location information of the prediction box.

[0060] Keypoint detection branch (i.e. Figure 1When training the Pose branch in the training, only the features corresponding to the real box are used for training. In the key point detection branch, for the predicted candidate boxes corresponding to the labeled boxes in all the cropped features, the corresponding features are passed through the ROI line 2 with an output feature size of 28×28 to obtain the proposa l features a of all Pose branches with a size of (n,64,28,28), and n is the number of all labeled boxes in a training batch. The feature corresponding to each proposa l in feature a is passed through the ordinate classification head in the Pose branch to obtain the ordinate value information of all types of points with a size of (c2,112). c2 is the total number of types of all points. The feature corresponding to each proposa l in feature a is passed through the abscissa classification head in the pose branch to obtain the abscissa value information of all types of points with a size of (c2,112). The feature corresponding to each proposa l in feature a is passed through the visibility classification head in the pose branch to obtain the visibility information of all types of points with a size of (c2,2).

[0061] We set a matching relationship between the box type and the point type in advance, as shown in Table 1.

[0062] Table 1. Matching relationship between various types of boxes and points

[0063] Box Type Point Type 0 0_1,0_2 1 1_1,1_2 2 2_1,2_2 3 3_1,3_2 4 4_1,4_2 5 5_1,5_2

[0064] When predicting visibility, the visibility of various types of points that match the box type is set to visible, and the remaining points are set to invisible. Therefore, our algorithm can realize the function of capturing specific types of points with specific types of boxes.

[0065] The network flow of the horizontal axis classification head, the vertical axis classification head and the visibility classification head is as follows Figure 2 As shown. Conv2D_y is a convolution with a horizontal step size of 2 and a vertical step size of 1, and Conv2D_x is a convolution with a horizontal step size of 1 and a vertical step size of 2. When the Pose branch is predicted, the input of the Pose branch is no longer the feature corresponding to the real box proposa l, but the feature corresponding to the box proposa l predicted by the Faster RCNN branch. The features of size (c2,112) output by the horizontal and vertical classification heads in the pose branch are passed through the Soft-argmax function to obtain the horizontal and vertical coordinate values ​​of all types of points. Finally, the type and coordinates of the final predicted points can be obtained by filtering the non-existent points through the network prediction visibility.

[0066] Training process:

[0067] 1. Labeled data: Different targets are labeled with different types of boxes, and various points corresponding to the boxes are marked inside the boxes, such as Figure 3 shown.

[0068] 2. Data preprocessing: The same data preprocessing as Faster RCNN is used, but the image midpoint is also processed accordingly in the preprocessing of changing the image size and position.

[0069] 3. Use the above method to train and predict images. The visualization effect is as follows Figure 4 , Figure 5 shown.

[0070] In summary, the technical effects that can be achieved by the present invention in Example 1 are:

[0071] 1. Propose a solution for measuring the size of industrial parts to solve the problem that the size of industrial parts is difficult to detect in real time under high-speed production.

[0072] 2. An end-to-end edge-grabbing algorithm based on AI detection is proposed to solve the problem that the AI ​​edge-grabbing algorithm that detects the target first and then the key points requires two training data sets, two models, and the key point detection model needs to be run multiple times.

[0073] 3. Design a new AI model for simultaneous target detection and key point detection. Add a key point detection branch based on Faster RCNN to realize the function of detecting specific types of points in different detection frames.

[0074] The above is an example of the best implementation of the present invention, and the parts not described in detail are common knowledge of ordinary technicians in the field. The protection scope of the present invention shall be based on the content of the claims, and any equivalent transformation based on the technical enlightenment of the present invention is also within the protection scope of the present invention.

Claims

1. An industrial dimension measurement method, characterized in that: The following steps are involved: Collect images of objects to be inspected; Extract multi-scale features in the image and predict candidate boxes corresponding to the multi-scale features; The method for extracting multi-scale features in an image includes: inputting the image into a feature extraction network and a feature pyramid network to obtain multi-scale features; The method for extracting a prediction candidate box corresponding to a multi-scale feature includes: passing the multi-scale feature through an RPN network to obtain a prediction candidate box; Crop the features corresponding to the predicted candidate box. After cropping the features corresponding to the predicted candidate box, the cropped features are uniformly scaled to a size of 7×7 through ROI Aline1. After the scaled features are flattened, the type and position information of each predicted candidate box is obtained through a fully connected network; Before all the cropped features are input into the key point detection branch, the key point detection branch is trained by collecting multiple image samples, selecting features in the image samples through real frame labels, obtaining a feature training set, and inputting the feature training set into the key point detection branch. After the training is completed, a trained key point detection branch is obtained; All the cropped features are input into the key point detection branch to obtain the horizontal coordinate value information, vertical coordinate value information and visibility information of all types of points in each feature; The key point detection branch includes the following steps: All cropped features are uniformly scaled to 28×28 size through 28×28 ROI Aline2, and all features a of size (n, 64, 28, 28) are obtained, where n is the number of features in all predicted candidate boxes; Input all features in feature a into the ordinate classification head to obtain the ordinate value information of all types of points of size (c2,112), where c2 is the total number of types of all points and 112 is 28×4 coordinate information; Input all features in feature a into the horizontal coordinate classification head to obtain the horizontal coordinate value information of all types of points of size (c2,112), where c2 is the total number of types of all points and 112 is 28×4 coordinate information; Input all features in feature a into the visibility classification head to obtain visibility information of all types of points of size (c2, 2), where c2 is the total number of types of all points and 2 is the visibility state of the point; After the features in feature a are input into the ordinate classification head, the following steps are included: After the features in feature a are convolved by Conv2D_y and activated by ReLU, the image feature y1 is obtained; After the feature y1 passes through the Conv2D_y convolution and ReLU activation function, the image feature y2 is obtained; After flattening feature y2, we get feature y3; After the feature y3 passes through one-dimensional deconvolution and ReLU activation function, feature y4 is obtained; After the feature y4 passes through the one-dimensional convolution and ReLU activation function, the vertical coordinate value information of the type point is obtained; After the features in feature a are input into the abscissa classification head, the following steps are included: After the features in feature a are convolved by Conv2D_x and activated by ReLU, image feature x1 is obtained; After the feature x1 passes through the Conv2D_x convolution and the ReLU activation function, the image feature x2 is obtained; After flattening feature x2, we get feature x3; After the feature x3 passes through one-dimensional deconvolution and ReLU activation function, feature x4 is obtained; After the feature x4 passes through the one-dimensional convolution and ReLU activation function, the horizontal coordinate value information of the type point is obtained; After the features in feature a are input into the visibility classification head, the following steps are included: After the features in feature a are subjected to two Conv2D convolutions and ReLU activation functions, and then globally pooled, flattened, and fully connected, the visibility information of the type point is obtained; The type of the predicted candidate box is matched with the type point, and the non-existent points are filtered out through network prediction visibility to obtain the type and coordinates of the final predicted point.

2. The industrial dimension measuring method according to claim 1, characterized in that: After the horizontal and vertical coordinate information of all types of points are processed by the Soft-argmax function, the coordinates of all types of points are obtained.

3. The industrial dimension measuring method according to claim 1, characterized in that: The method for matching the type of the predicted candidate box with the type point includes the following steps: Set the matching relationship between the box type and the point type; Sets the visibility of type points that match the box type to Visible, and points that do not match to Invisible.

Images