Lightweight industrial image key point detection method based on classification model

Abstract

The application relates to a lightweight industrial image key point detection method based on a classification model, which comprises the following steps: inputting a to-be-detected image into a feature extraction network, extracting image features of key points through the feature extraction network; inputting the image features into a network output head, outputting predicted horizontal coordinates, predicted vertical coordinates and predicted visibility of the key points through the network output head; labeling the key points on the to-be-detected image, mapping the key points to a horizontal coordinate axis and a vertical coordinate axis, and calculating a first loss value of the horizontal coordinates of the key points and the predicted horizontal coordinates, a second loss value of the vertical coordinates and the predicted vertical coordinates, and a third loss value of real visibility information and the predicted visibility. The application can meet the real-time operation requirements of low-computing-power equipment, eliminate the theoretical error lower bound based on a heat map model based on the classification model, and avoid detecting non-existing key points by increasing the visibility loss function of the key points.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image key point detection, in particular to a lightweight industrial image key point detection method based on a classification model. BACKGROUND

[0002] Industrial key point detection refers to automatically identifying the key point positions in industrial products from images or videos in industrial scenes, such as the end effector of a robot, the positioning point of a part, etc. Traditional industrial key point detection algorithms require a large amount of computing resources and often need to train large-scale data sets, so there are certain limitations in actual application. Therefore, there is an urgent need for lightweight industrial key point detection algorithms to achieve better key point detection results in the case of limited computing resources.

[0003] The model based on heat map representation needs multiple high-overhead deconvolutions to generate heat maps, so it is slow and high-overhead. Since the heat map is several times smaller than the original image, the method based on heat map will have a lower bound of theoretical error. The regression-based method is greatly affected by noise and randomness, resulting in unstable results. The classification-based model does not generate heat maps, and the dimension of classification can be the size of the original image, with higher accuracy. The classification model is less affected by noise and randomness, and the training process is more stable, and the model is more easily converged. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a lightweight industrial image key point detection method based on a classification model to solve the above problems.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] The lightweight industrial image key point detection method based on the classification model comprises the following steps:

[0007] Input the image to be detected into the feature extraction network, and extract the image features of the key points through the feature extraction network;

[0008] Input the image features into the network output head, and output the predicted horizontal coordinates, predicted vertical coordinates and predicted visibility of the key points through the network output head;

[0009] Label the key points on the image to be detected, map the key points to the horizontal and vertical coordinate axes, calculate the first loss value of the horizontal coordinates of the key points and the predicted horizontal coordinates, the second loss value of the vertical coordinates and the predicted vertical coordinates, and the third loss value of the real visibility information and the predicted visibility, and according to the total loss value of the first loss value, the second loss value and the third loss value, judge whether the total loss value can pass the threshold value, if not, filter out the key points.

[0010] Further, the feature extraction network comprises one layer of MobileNetV3 small and a convolution with a step size of 1.

[0011] Further, the network output head comprises a longitudinal coordinate classification head, a transverse coordinate classification head, and a visibility classification head, the longitudinal coordinate classification head predicts the longitudinal coordinate of the key point, the transverse coordinate classification head predicts the transverse coordinate of the key point, and the visibility classification head predicts the visibility of the key point.

[0012] Further, the longitudinal coordinate classification head predicting the longitudinal coordinate of the key point comprises the following steps:

[0013] After the image features are subjected to two-dimensional convolution with a step size of 2 in the transverse coordinate direction and a step size of 1 in the longitudinal coordinate direction and ReLU activation function, image features y1 are obtained;

[0014] After the features y1 are subjected to two-dimensional convolution with a step size of 2 in the transverse coordinate direction and a step size of 1 in the longitudinal coordinate direction and ReLU activation function, image features y2 are obtained;

[0015] After the features y2 in the transverse coordinate direction are subjected to global pooling, features y3 are obtained;

[0016] After the features y3 are subjected to one-dimensional deconvolution, features y4 are obtained;

[0017] After the features y4 are subjected to one-dimensional deconvolution and ReLU activation function, the predicted longitudinal coordinate of the key point is obtained.

[0018] Further, the transverse coordinate classification head predicting the transverse coordinate of the key point comprises the following steps:

[0019] After the image features are subjected to two-dimensional convolution with a step size of 1 in the transverse coordinate direction and a step size of 2 in the longitudinal coordinate direction and ReLU activation function, image features x1 are obtained;

[0020] After the features x1 are subjected to two-dimensional convolution with a step size of 1 in the transverse coordinate direction and a step size of 2 in the longitudinal coordinate direction and ReLU activation function, image features x2 are obtained;

[0021] After the features x2 in the longitudinal coordinate direction are subjected to global pooling, features x3 are obtained;

[0022] After the features x3 are subjected to one-dimensional deconvolution, features x4 are obtained;

[0023] After the features x4 are subjected to one-dimensional deconvolution and ReLU activation function, the predicted transverse coordinate of the key point is obtained.

[0024] Further, the visibility classification head predicting the visibility of the key point comprises the following steps:

[0025] The image features are subjected to twice convolution in the horizontal coordinate direction and the vertical coordinate direction, with a size of 4, so that the length and width of the image features become 1 / 64 of the original, and then the image features after convolution are subjected to globalization and full connection operation to obtain the predicted visibility of the key points.

[0026] Further, the first loss value of the horizontal coordinate of the key point and the predicted horizontal coordinate and the second loss value of the vertical coordinate and the predicted vertical coordinate are calculated, including the following steps:

[0027] After the key points are mapped to the horizontal coordinate axis and the vertical coordinate axis, one-hot encoding in the horizontal coordinate direction and one-hot encoding in the vertical coordinate direction are formed, and then the first loss value of the one-hot encoding in the horizontal coordinate direction and the predicted horizontal coordinate and the second loss value of the one-hot encoding in the vertical coordinate direction and the predicted vertical coordinate are calculated by using a cross-entropy loss function.

[0028] Further, the method for calculating the third loss value of the real visibility information of the key points and the predicted visibility includes:

[0029] The third loss value of the real visibility information of the key points and the predicted visibility is calculated by using a binary cross-entropy loss function.

[0030] Further, the total loss value is the weighted sum of the first loss value, the second loss value and the third loss value, and the weights of the first loss value, the second loss value and the third loss value are determined according to the prediction difficulty of the horizontal coordinate, the vertical coordinate and the visibility, and the prediction difficulty is proportional to the weight.

[0031] After the above technical scheme is adopted in the present application, the following advantages are achieved compared with the prior art:

[0032] The industrial image key point detection algorithm based on the regression model can automatically detect various types of key points in the industrial image, and the average pixel error of various types of key points reaches a sub-pixel level, the precision is higher than that of the traditional detection algorithm, the subjective factors caused by individual differences, body state and the like are eliminated, the safety, reliability and efficiency of detection are improved, the calculation resources required by the key point detection algorithm are reduced, the algorithm can run on a low-power device, the visibility loss function of the key point is added, the position information of each type of point is outputted, and the visibility of the point is also outputted, the non-existing points (i.e. the points that cannot pass the threshold or are not in the image range) can be screened, and the over-detection rate of the classification model is reduced.

[0033] The present application will be described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 It is a schematic diagram of the overall process of the present application.

[0035] Figure 2 A network flow schematic diagram of the longitudinal coordinate classification head of the present application;

[0036] Figure 3 A network flow schematic diagram of the transverse coordinate classification head of the present application;

[0037] Figure 4 A network flow schematic diagram of the visibility classification head of the present application. DETAILED DESCRIPTION

[0038] The principles and features of the present application are described below in conjunction with the accompanying drawings, which are only used to explain the present application and are not intended to limit the scope of the present application.

[0039] As shown in Figure 1 The lightweight industrial image key point detection method based on a classification model comprises the following steps:

[0040] The image to be detected is input into a feature extraction network, and the image features of the key points are extracted through the feature extraction network;

[0041] The image features are input into a network output head, the network output head comprising a longitudinal coordinate classification head, a transverse coordinate classification head and a visibility classification head, the longitudinal coordinate classification head predicting the longitudinal coordinates of the key points, the transverse coordinate classification head predicting the transverse coordinates of the key points, and the visibility classification head predicting the visibility of the key points, and the predicted transverse coordinates, predicted longitudinal coordinates and predicted visibility of the key points are output through the network output head;

[0042] The key points are labeled on the image to be detected, and the key points are mapped onto the transverse coordinate axis and the longitudinal coordinate axis, the first loss value of the transverse coordinates of the key points and the predicted transverse coordinates, the second loss value of the longitudinal coordinates and the predicted longitudinal coordinates, and the third loss value of the real visibility information and the predicted visibility are calculated, the total loss value is calculated according to the first loss value, the second loss value and the third loss value, and it is determined whether the total loss value can pass the threshold value, and if not, the key points are filtered out.

[0043] An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the steps of the above-mentioned lightweight industrial image key point detection method based on a classification model.

[0044] A non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the steps of the above-mentioned lightweight industrial image key point detection method based on a classification model.

[0045] Example 1:

[0046] Traditional industrial key point detection algorithms require a large amount of computing resources, and often need to train large data sets, so there are certain limitations in practical applications. The key point detection algorithm based on heat map representation runs slowly, has high overhead, and has a theoretical error lower bound. The heat map has a theoretical error lower bound of several pixels after several downsampling of the original image. The general key point classification model will output all types of points, regardless of whether a certain type of point exists in the image. Points that do not exist in the image will also be detected. Therefore, the present application proposes a lightweight key point detection algorithm to meet the real-time operation requirements of low-power devices, and eliminates the theoretical error lower bound of the heat map model based on the classification model. By increasing the visibility loss function of the key point, the visibility of the key point is judged to avoid detecting non-existent key points.

[0047] I. Feature extraction network: As shown in Figure 1 , the feature extraction network is composed of a convolution with a step of 2 and a layer of inverted residual structure in MobileNetV3 small. MobileNetV3 small has 11 layers of inverted residual structure. In order to speed up the inference and reduce the parameter amount, we only apply one layer for feature extraction.

[0048] II. Network output head: The network structure is as shown in Figure 1 , the output head is composed of three parts, which are the vertical coordinate classification head for predicting the vertical coordinate of the key point, the horizontal coordinate classification head for predicting the horizontal coordinate of the key point, and the visibility classification head for predicting the visibility of the key point.

[0049] The vertical coordinate classification head is as shown in Figure 2 , the feature extraction network extracts the features of the image, which first passes through a two-dimensional convolution with a horizontal coordinate direction step of 2 and a vertical coordinate direction step of 1 and a ReLU activation function to obtain an image feature y1 with a feature map width reduced by half and a height unchanged, and then uses the same convolution to obtain an image feature y2. The features in the horizontal coordinate direction of the image feature y2 are globally pooled to obtain an image feature y3. The image feature y3 is one-dimensional deconvolved to change the image feature dimension to a times the height of the original image to obtain an image feature y4, which aims to make the positioning accuracy of the key point reach the sub-pixel level. Finally, the image feature y4 is one-dimensional convolved and ReLU activated to obtain the output of the vertical coordinate classification head, that is, the predicted vertical coordinate of the key point. Similarly, the output of the horizontal coordinate classification head can be obtained, and the process is as shown in Figure 3 .

[0050] The model of the present application can not only obtain the detailed position of the key point, but also predict the visibility of each type of key point. That is, it can distinguish between points that exist in the original image and points that do not exist. The visibility branch of the network (visibility classification head) is as shown in Figure 4As shown, the image features are first subjected to twice convolution in horizontal and vertical directions with a step size of 4 to change the feature length and width to 1 / 64 of the original image. Then, after global pooling, flattening, and full connection operations, the output of the network visibility branch is obtained.

[0051] III. Loss function: First, the key points labeled in the image (the key points labeled in the image are obtained by manual labeling on the image) are mapped to two coordinate axes to form one-hot encoding. The cross-entropy loss function is used to calculate the loss function value of the one-hot encoding in two directions of each point, the output of the network vertical coordinate output head, and the output of the horizontal coordinate output head. The binary cross-entropy loss function is used to calculate the loss function value of the real visibility information of each point and the output of the network visibility output head. The total loss function is the weighted sum of the three loss functions. The weight is determined according to the prediction difficulty (attention) of the points in two directions (the prediction difficulty is determined according to the actual prediction situation, and the weight of the point with greater prediction difficulty is higher). For example, if the prediction difficulty (attention) of the horizontal coordinate position is greater, the weights of the first loss value, the second loss value, and the third loss value are set to 0.5, 0.25, and 0.25, respectively. If the prediction difficulty (attention) of the vertical coordinate position is greater, the weights of the first loss value, the second loss value, and the third loss value are set to 0.25, 0.5, and 0.25, respectively.

[0052] The industrial image key point detection algorithm based on the regression model of the present application can automatically detect various types of key points in industrial images, and the average pixel error of various key points can reach sub-pixel level, which has higher precision than traditional detection algorithms, and eliminates the influence of subjective factors such as individual differences and body state on defect judgment caused by manual work, improves the safety, reliability and efficiency of detection. Compared with common key point classification models, the present application has the function of predicting point visibility, which greatly reduces the average over-detection rate of common key point regression models.

[0053] The above describes the best embodiment of the present application, wherein parts not described in detail are common knowledge of ordinary skilled in the art. The protection scope of the present application is subject to the content of the claims, and any equivalent transformation based on the technical inspiration of the present application is also within the protection scope of the present application.

Claims

1. A lightweight industrial image key point detection method based on a classification model, characterized in that, Includes the following steps: The image to be detected is input into a feature extraction network, which includes a convolution with a stride of 1 and a single inverse residual structure in MobileNetV3 small; the image features of key points are extracted through the feature extraction network. Image features are input into the network output head, which includes a ordinate classification head, an abscissa classification head, and a visibility classification head. The network output head outputs the predicted abscissa, predicted ordinate, and predicted visibility of key points. The ordinate classification head predicts the ordinate of the key points, the abscissa classification head predicts the abscissa of the key points, and the visibility classification head predicts the visibility of the key points. The method of predicting the ordinate of key points using the ordinate classification head includes the following steps: The image features are processed by a two-dimensional convolution with a stride of 2 in the horizontal direction and a stride of 1 in the vertical direction, and then activated by the ReLU function to obtain the image features y1. After passing feature y1 through a two-dimensional convolution with a stride of 2 in the horizontal direction and a stride of 1 in the vertical direction, and then through a ReLU activation function, image feature y2 is obtained. After performing global pooling on the features along the horizontal axis of feature y2, we obtain feature y3; After performing a one-dimensional deconvolution on feature y3, feature y4 is obtained; After passing feature y4 through one-dimensional deconvolution and ReLU activation function, the predicted ordinate of the key point is obtained; The horizontal coordinate classification head predicts the horizontal coordinates of key points by including the following steps: The image features are processed by a two-dimensional convolution with a stride of 1 in the horizontal direction and a stride of 2 in the vertical direction, followed by a ReLU activation function, to obtain image feature x1. After passing feature x1 through a two-dimensional convolution with a stride of 1 in the horizontal direction and a stride of 2 in the vertical direction, and then through a ReLU activation function, we obtain image feature x2. After performing global pooling on the features along the vertical axis of feature x2, we obtain feature x3; After performing a one-dimensional deconvolution on feature x3, feature x4 is obtained; After passing feature x4 through one-dimensional deconvolution and ReLU activation function, the predicted x-coordinate of the keypoint is obtained; the visibility classification head predicts the visibility of keypoints through the following steps: The image features are convolved twice with 4 in both the horizontal and vertical directions, making the length and width of the image features 1 / 64 of the original. After the convolved image features are globalized and fully connected, the predictive visibility of the key points is obtained. Key points are marked on the image to be detected and mapped onto the horizontal and vertical axes. The first loss value between the horizontal coordinate and the predicted horizontal coordinate, the second loss value between the vertical coordinate and the predicted vertical coordinate, and the third loss value between the true visibility information and the predicted visibility are calculated. Based on the total loss value of the first, second, and third loss values, it is determined whether the total loss value can pass the threshold. If not, the key point is filtered out.

2. The lightweight industrial image key point detection method based on a classification model according to claim 1, characterized in that, Calculate the first loss value (comparing the x-coordinate to the predicted x-coordinate) and the second loss value (comparing the y-coordinate to the predicted y-coordinate) of the keypoints, including the following steps: After mapping keypoints to the horizontal and vertical axes, one-hot codes are formed in the horizontal direction and the vertical direction. Then, the cross-entropy loss function is used to calculate the first loss value of the one-hot code in the horizontal direction and the first loss value of the one-hot code in the vertical direction and the second loss value of the one-hot code in the vertical direction and the first loss value of the one-hot code in the vertical direction.

3. The lightweight industrial image key point detection method based on a classification model according to claim 1, characterized in that, Methods for calculating the third loss value between the true visibility information and the predicted visibility of key points include: The binary cross-entropy loss function is used to calculate a third loss value for the true visibility information and predicted visibility of key points.

4. The lightweight industrial image key point detection method based on a classification model according to claim 1, characterized in that, The total loss value is the weighted sum of the first loss value, the second loss value, and the third loss value. The weights of the first loss value, the second loss value, and the third loss value are determined based on the prediction difficulty of the horizontal axis, the vertical axis, and the visibility, respectively. The prediction difficulty is proportional to the weight.

Images