Workpiece counting method and device and medium

Abstract

The invention provides a workpiece counting method and device and a medium. Cutting the workpiece image to obtain a slice image; performing workpiece detection according to the workpiece image through a detector of a workpiece classification model to obtain a first detection image containing a first prediction frame, and performing workpiece detection according to the slice image to obtain a second detection image containing a second prediction frame; combining the first detection image and the second detection image, and selecting a target prediction frame from the first prediction frame and the second prediction frame to obtain a combined detection image; counting the combined detection image according to clustering to obtain the number of various workpieces; according to the method, a proper prediction frame can be screened from different prediction frames, proper features can be extracted from limited and distorted sample information, and the accuracy of classification and counting is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a workpiece counting method, device and medium. Background Art

[0002] In the process of workpiece production, the number of workpieces needs to be counted for subsequent material management processes such as warehouse management, logistics transportation, sales orders, etc. The workpiece counting method based on deep learning often faces limitations from data conditions, such as insufficient number of samples of a certain type, imbalanced number; or the quality of the data set itself has problems such as incorrect labeling and unreasonable category settings. At the same time, the inherent structure of low boundary discrimination targets causes blurred visual appearance and background noise interference. The low-quality feature representation of low boundary discrimination targets can be attributed to their limited size and general feature extraction paradigm; in small target detection tasks, feature maps are usually downsampled to eliminate spatial redundancy and learn high-dimensional features, which will damage the feature representation of small objects. This makes it difficult to extract suitable features from limited and distorted sample information, greatly reducing the accuracy of workpiece classification and counting. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of the present application is to solve one of the technical problems existing in the related art to at least a certain extent. The embodiments of the present application provide a workpiece counting method, device and medium to improve the accuracy of workpiece classification and counting.

[0005] An embodiment of the first aspect of the present application, a workpiece counting method, comprises:

[0006] Acquire a workpiece image, and input the workpiece image into a workpiece classification model;

[0007] Cutting the workpiece image to obtain a slice image;

[0008] Performing workpiece detection according to the workpiece image by a detector of the workpiece classification model to obtain a first detection image including a first prediction frame, and performing workpiece detection according to the slice image to obtain a second detection image including a second prediction frame;

[0009] Merging the first detection image and the second detection image, and selecting a target prediction frame from the first prediction frame and the second prediction frame to obtain a merged detection image;

[0010] The combined detection image is counted according to clustering to obtain the number of each type of workpiece.

[0011] According to certain embodiments of the first aspect of the present application, the step of cutting the workpiece image to obtain a slice image includes:

[0012] The workpiece image is cut according to a preset step length to obtain a plurality of partially overlapping slice images.

[0013] According to certain embodiments of the first aspect of the present application, performing workpiece detection by a detector of the workpiece classification model includes:

[0014] Perform feature extraction on the input image to obtain a first feature;

[0015] Position-encode the first feature and map it to a low-dimensional space to obtain a coded feature;

[0016] Decoding the encoded features to obtain decoded features;

[0017] Perform detection according to the decoded features to obtain a detection image including a class and a prediction box;

[0018] Among them, when the input image is the workpiece image, the prediction box is the first prediction box, and the detection image is the first detection image; when the input image is the slice image, the prediction box is the second prediction box, and the detection image is the second detection image.

[0019] According to certain embodiments of the first aspect of the present application, decoding the encoded feature to obtain a decoded feature includes:

[0020] Reshaping the encoded feature in width and height to obtain a reshaped feature;

[0021] Upsampling the reshaped features to obtain upsampled features;

[0022] Convolving a fusion result of the up-sampled feature and the encoded feature to obtain a convolution feature, and adding the convolution feature and the up-sampled feature to obtain a second feature;

[0023] The second feature and the sum of the feature extraction results of the second feature are pooled to obtain a decoding feature.

[0024] According to certain embodiments of the first aspect of the present application, the detecting according to the decoded features to obtain a detection image including a class and a prediction box includes:

[0025] Detecting according to the decoded features to obtain a candidate prediction box;

[0026] Determine a first distance between a center of a bounding box of the workpiece image and the candidate prediction box;

[0027] When the difference between the intersection-over-union ratio of the candidate prediction box and the bounding box and the first distance is less than a distance threshold, selecting the candidate prediction box as the prediction box;

[0028] Classification is performed according to the candidate boxes to obtain classes, and then a detection image including the classes and the prediction boxes is obtained.

[0029] According to certain embodiments of the first aspect of the present application, the detecting according to the decoding feature to obtain a candidate prediction frame includes:

[0030] Regressing the distance between the feature point of the decoded feature and the boundary of the workpiece image to obtain a bounding box;

[0031] Allocating the bounding box to a corresponding prediction layer according to a workpiece scale in the bounding box, wherein the workpiece scale is determined according to a distance between a center point of the bounding box and a boundary of the workpiece image;

[0032] Determine a positive sample area and a negative sample area according to the center point of the bounding box;

[0033] Classifying the anchor frame as a positive sample or a negative sample according to the overlap between the anchor frame and the positive sample region and the overlap between the anchor frame and the negative sample region;

[0034] Training the prediction layer according to the positive sample and the negative sample to obtain a training prediction box, and adjusting the parameters of the prediction layer according to the distance between the training prediction box and the center point of the bounding box to obtain a trained prediction layer;

[0035] The candidate prediction box is obtained by detecting the decoded features through the trained prediction layer.

[0036] According to some embodiments of the first aspect of the present application, the selecting a target prediction box from the first prediction box and the second prediction box includes:

[0037] A target prediction box is selected from the first prediction box and the second prediction box according to a numerical relationship between an area value of the first prediction box, an area value of the second prediction box and a prediction area threshold.

[0038] According to certain embodiments of the first aspect of the present application, counting the combined detection image according to clustering to obtain the number of each type of artifacts includes:

[0039] The number of target categories is determined by calculating the ratio of the distance between clusters and the distance within the cluster according to the combined detection image;

[0040] The intra-class distance and the inter-class distance are calculated according to the number of target categories, and the number of each category of workpieces is obtained according to the intra-class distance and the inter-class distance.

[0041] An embodiment of the second aspect of the present application is an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the workpiece counting method as described in the embodiment of the first aspect of the present application is implemented.

[0042] An embodiment of the third aspect of the present application is a computer storage medium, characterized in that it stores computer executable instructions, and the computer executable instructions are used to execute the workpiece counting method as described in the embodiment of the first aspect of the present application.

[0043] The above scheme has at least the following beneficial effects: a slice image is obtained by cropping the workpiece image; a detector of the workpiece classification model performs workpiece detection based on the workpiece image to obtain a first detection image including a first prediction frame, and performs workpiece detection based on the slice image to obtain a second detection image including a second prediction frame; the first detection image and the second detection image are merged, and a target prediction frame is selected from the first prediction frame and the second prediction frame to obtain a merged detection image; the merged detection image is counted according to clustering to obtain the number of each type of workpiece; a suitable prediction frame can be screened out from different prediction frames, and suitable features can be extracted from limited and distorted sample information, thereby improving the accuracy of classification and counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0045] Figure 1 is a step diagram of a workpiece counting method provided in an embodiment of the present application;

[0046] Figure 2 is a diagram of steps for performing workpiece detection by a detector of a workpiece classification model provided by an embodiment of the present application;

[0047] Figure 3 It is a step diagram of decoding the coded feature by a decoder to obtain the decoded feature provided by an embodiment of the present application;

[0048] Figure 4 It is a step diagram of performing detection according to the decoded features to obtain a detection image including a class and a prediction box provided by an embodiment of the present application;

[0049] Figure 5 It is a step diagram of obtaining a candidate prediction frame by detecting according to the decoding features provided in an embodiment of the present application;

[0050] Figure 6It is a step diagram of counting the merged detection image according to clustering to obtain the number of each type of workpiece provided by an embodiment of the present application;

[0051] Figure 7 is a structural diagram of a backbone network, an encoder, and a decoder provided in an embodiment of the present application;

[0052] Figure 8 is a structural diagram of a decoder provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the corresponding step S400 provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0057] An embodiment of the present application provides a workpiece counting method.

[0058] Reference Figure 1 , the workpiece counting method comprises the following steps:

[0059] Step S100, acquiring a workpiece image, and inputting the workpiece image into a workpiece classification model;

[0060] Step S200, cutting the workpiece image to obtain a slice image;

[0061] Step S300, performing workpiece detection on the workpiece image by a detector of the workpiece classification model to obtain a first detection image including a first prediction frame, and performing workpiece detection on the slice image to obtain a second detection image including a second prediction frame;

[0062] Step S400, merging the first detection image and the second detection image, and selecting a target prediction frame from the first prediction frame and the second prediction frame to obtain a merged detection image;

[0063] Step S500: counting the combined detection image according to clustering to obtain the number of each type of workpiece.

[0064] For step S100, the workpiece image is obtained. On the one hand, the workpiece image is collected from the storage rack containing the workpiece to be detected in the production line and warehouse in the industrial scene by an industrial camera, and the data collection of the workpiece under different lighting, background, and scene shielding is considered. On the other hand, the image acquisition mode is turned on by the camera of the mobile device, and the target workpiece on the warehouse rack is photographed, or the workpiece image is directly imported from the album of the mobile device.

[0065] Images with high similarity and junk data are cleaned, workpiece images are cropped and adjusted to a fixed size, and a dense target dataset of industrial images is constructed to enhance the stability of model training.

[0066] For the preprocessed images, the three-dimensional thinking at the human level is integrated to construct the dataset to meet the needs of real industrial scenarios. In order to meet the human-level three-dimensional thinking analysis in the interpretation dataset: how does the visual network identify devices from one or several new examples (Level-1); how does the algorithm observe such occluded, ultra-small objects and low-visibility devices (Level-2); how does the algorithm parse the stacked devices into the relationship between parts and the whole according to the existing categories (Level-3); in daily device counting, the above problems often do not appear alone. When they appear collectively, they will get a greater challenge: how can the algorithm successfully identify from such sparse device data, while learning such rich representations to parse the stacking style of low-visibility devices (Level-K); the dataset is further divided into multiple different levels of validation sets, and Level-K is verified on different baseline algorithms. In order to train and evaluate the model, the dataset is divided into a training set (Train) for training the model and a test set (Test) for proving the generalization ability of the model. The two databases need to meet independent and identical distribution. In addition, a part of the data needs to be extracted from the training set as a validation set (Val). Therefore, the training set, validation set, and test set are divided in a ratio of 5:2:3.

[0067] For example, the dataset of artifact images contains a total of 200 multi-class artifact classification scenes, including 100 scenes for two-class artifact classification and another 100 scenes for three-class artifact classification.

[0068] A dataset of artifact images is input to an artifact classification model.

[0069] For step S200, for dense and complex stacking scenarios, a slice-assisted training and fine-tuning pipeline is introduced for complex and dense target detector training. The workpiece image is cropped to obtain a slice image, making the model more suitable for dense detection tasks in industrial environments.

[0070] The workpiece image is cut to obtain a slice image, comprising the following steps: cutting the workpiece image according to a preset step length to obtain a plurality of partially overlapping slice images

[0071] During the training phase, the input image is divided into overlapping patches of a certain area. The pixel area of ​​complex and difficult-to-distinguish boundary objects will be relatively small compared to the image input to the network. First, the original query image I is divided into L M×N overlapping patches. Then, each patch is resized while maintaining the aspect ratio. Each overlapping patch is then independently fed into the object detection network for forward propagation training.

[0072] In the device detection task of images taken in industrial scenes, the accuracy of dense target stacked device images is significantly lower. Compared with training from scratch using a large dataset, this allows the model to be fine-tuned using a smaller dataset and a shorter training span. Most of these common datasets involve low-resolution images (e.g., 640×480 images) with fairly large objects with large pixel coverage, which covers an average of 60% of the image height. Models pre-trained using these datasets provide very successful detection performance for similar inputs.

[0073] In the inference phase, the image segmentation method is the same as that in the training phase.

[0074] In step S300, the detector of the workpiece classification model performs workpiece detection on the workpiece image to obtain a first detection image including a first prediction frame, and performs workpiece detection on the slice image to obtain a second detection image including a second prediction frame.

[0075] Reference Figure 2 , artifact detection is performed by the detector of the artifact classification model, including the following steps:

[0076] Step S310, extracting features from the input image to obtain a first feature;

[0077] Step S320, position encoding the first feature and mapping it to a low-dimensional space to obtain a coded feature;

[0078] Step S330, decoding the encoded feature to obtain a decoded feature;

[0079] Step S340, performing detection according to the decoded features to obtain a detection image including the class and the prediction box;

[0080] Among them, when the input image is a workpiece image, the output is a first detection image containing a class and a first prediction box; when the input image is a slice image, the output is a second detection image containing a class and a second prediction box.

[0081] The backbone network extracts features from the input image to obtain the first feature. The workpiece image or slice image is converted into a feature map for subsequent classification prediction through a series of feature extraction layers. Darknet53 is used as the backbone network. Darknet53 uses residual connection and batch normalization technologies, which have great advantages in improving the convergence speed and performance of the model. The main features of Darknet53 are the large number of network layers and the large number of parameters, which make the model have strong feature extraction capabilities.

[0082] The encoder encodes the first feature by position and maps it to a low-dimensional space to obtain the encoded feature. The encoder is mainly used to position encode and embed the feature map and map the input data to a low-dimensional space, retaining its main features while removing irrelevant noise and redundant information for decoding prediction by the decoder. First, a 1×1 convolution is used to reduce the channel dimension of the input sequence from C to d, and the first feature Z0∈R is obtained. d×H×W . The first feature is fed into a stacked Transformer encoder layer, which mainly consists of a multi-head self-attention and a feedforward network to mix spatial and channel information, respectively. The encoder is constructed using ConvNeXt blocks. Specifically, each encoder layer consists of a 7×7Depthwise convolution, a LayerNorm layer, a 1×1 convolution, a GELU activation, and another 1×1 convolution stack. In order to align the FLOPs with DETR, the encoder consists of three stages with the number of blocks being (2, 6, 2) and the channel size being (120, 240, 480). Position encoding is added to the input of each Transformer encoder layer. Since the Transformer architecture is permutation invariant and the convolutional network structure is permutation variant, the decoder can get rid of the position encoding.

[0083] Reference Figure 3 , decoding the encoded features through a decoder to obtain decoded features, including the following steps:

[0084] Step S331, reshape the coded feature in width and height to obtain a reshaped feature;

[0085] Step S332, upsampling the reshaped features to obtain upsampled features;

[0086] Step S333, convolving the fusion result of the up-sampled feature and the encoded feature to obtain a convolution feature, and adding the convolution feature and the up-sampled feature to obtain a second feature;

[0087] Step S334, performing pooling processing on the second feature and the sum of the feature extraction results of the second feature to obtain a decoding feature.

[0088] Reference Figure 8 , the decoder aims to reason about the relationship between objects and global image features. The feature map sequence and the localization sequence are taken as the input of the decoder. Each layer of the decoder mainly consists of a self-interaction module and a cross-interaction module. The self-interaction module is a multi-head self-attention layer responsible for the interactive information between object queries. The cross-interaction module consists of a cross-attention layer for interaction between the image embedding output by the encoder and the object query. In this way, the object query can focus on the global image features and capture the essential information of each predicted object.

[0089] For the self-interaction module, given N object positioning sequences o∈R N×d , first reshape the positioning sequence into and feed them into the convolutional layer. For example, if there are N = 100 object localization sequences, the localization sequence embedding is reshaped into R 10 ×10×d Stacking a deep convolution with two 1×1 convolutions can produce a powerful self-attention mechanism, using a large kernel convolution 9×9DWConv to perform long-range perception feature extraction.

[0090] For the cross-interaction module, two features are mainly used as input, and the image feature embedding Z output from the encoder e ∈R d×H×W and object query embeddings generated from the self-interaction module The criss-cross attention mechanism in the decoder allows each object query to interact with the image features to capture the information required for object prediction. However, it is not so intuitive to use a convolutional network to perform this interaction. The object query o needs to be upsampled first to obtain So that it has the same characteristics as the image z e The same dimension, upsampling is expressed as: Then, the upsampled object query and image feature embedding are fused together through the addition operation, and then a large kernel depth convolution 9×9DWConv is performed, so that the object query can obtain spatial information from the image features, which can be expressed by the following formula: The output features are further passed through another feed-forward network with skip connections. Finally, adaptive max pooling is used to downsample the object query back to The size can be expressed by the following formula:

[0091] Reference Figure 4 , detect according to the decoded features, and obtain the detection image containing the class and the prediction box, including the following steps:

[0092] Step S341, detecting and obtaining candidate prediction boxes according to the decoded features;

[0093] Step S342, determining a first distance between the center of the bounding box of the workpiece image and the candidate prediction box;

[0094] Step S343, when the difference between the intersection-over-union ratio of the candidate prediction box and the bounding box and the first distance is less than the distance threshold, the candidate prediction box is selected as the prediction box;

[0095] Step S344, classify the candidate boxes to obtain classes, and then obtain a detection image including the classes and the prediction boxes.

[0096] Reference Figure 5 , wherein detecting and obtaining a candidate prediction box according to the decoded features includes the following steps:

[0097] Step S3411, performing regression based on the distance between the feature point of the decoded feature and the boundary of the workpiece image to obtain a bounding box;

[0098] Step S3412, assigning the bounding box to a corresponding prediction layer according to the scale of the workpiece in the bounding box;

[0099] Step S3413, determining the positive sample area and the negative sample area according to the center point of the bounding box;

[0100] Step S3414, classifying the anchor frame as a positive sample or a negative sample according to the overlap between the anchor frame and the positive sample region and the overlap between the anchor frame and the negative sample region;

[0101] Step S3415, obtaining a training prediction frame according to the positive sample and the negative sample training prediction layer, and adjusting the parameters of the prediction layer according to the distance between the training prediction frame and the center point of the bounding box to obtain a trained prediction layer;

[0102] Step S3416, the candidate prediction box is obtained by detecting according to the decoded features through the trained prediction layer.

[0103] During the inference phase, the segmented images are independently fed into the target detection network for prediction, while the original images are used to detect larger objects using a full inference approach, retaining the inference on the original images.

[0104] Use a fully convolutional network to directly decode the feature points at each position of the feature to the original Figure 4 The vertical and horizontal distances (l, t, r, b) between the point and the ground truth are used to represent the bounding box, and the distribution of positive and negative samples is determined by introducing layers, regions, and centers.

[0105] The artifact scale is determined by the distance between the center point of the bounding box and the boundary of the artifact image. The maximum scale (Min_Size) and the minimum scale (Max_Size) are introduced, and the specific settings are 0, 64, 128, 256, 512 and infinity. For example, for the first prediction layer of the output, its step size is 8, which is responsible for the smallest scale object. For any point on this layer, if there is a bounding box mapped to the feature map, it satisfies 0<max(center point to original Figure 4 If the distance between the edges is less than 64, then the bounding box belongs to the first prediction layer, and the remaining layers are also responsible for the same principle. In summary, the first layer is responsible for predicting bounding boxes with scales ranging from 0 to 64, the second layer is responsible for predicting bounding boxes with scales ranging from 64 to 28, and so on. This allocation strategy can be used to assign bounding boxes of different sizes to the most appropriate prediction layer for learning.

[0106] Set the Center_Sample_Radius parameter to determine that all samples within the radius belong to the positive sample area, and the rest of the area is the negative sample. The default configuration is Center_Sample_Radius = 1.5. For example, the step size of the first layer is 8, then on the output layer, for any bounding box, based on the center point of the bounding box as the starting point, all points within a radius of 1.5×8=12 pixels belong to the positive sample area. According to the overlap between the anchor box and the positive sample area and the overlap between the anchor box and the negative sample area, the anchor box is classified as a positive sample or a negative sample.

[0107] The prediction layer is trained based on positive and negative samples to obtain a training prediction box, and the parameters of the prediction layer are adjusted based on the distance between the training prediction box and the center point of the bounding box. Specifically, Determine the center point of the bounding box, multiply the center score by the classification score when inferring, and penalize the scores of the training prediction boxes that deviate far from the target center, so that the closer to the target center, the larger the output value, and vice versa; adjust the parameters of the prediction layer according to the penalty score.

[0108] The candidate prediction boxes are obtained by detecting the decoded features through the trained prediction layer.

[0109] Since the network may output multiple different results for the same target in each slice image, each object may have repeated prediction boxes. Before merging them into the original image, the candidate boxes need to be screened by non-maximum suppression. Determine the first distance R between the center of the bounding box of the workpiece image and the candidate prediction box DIoU (M,B i ); When the difference between the intersection of the candidate prediction box and the bounding box and the first distance is less than the distance threshold, the candidate prediction box is selected as the prediction box. It can be expressed by the following formula: IoU-R DIoU (M,Bi )<ε; where ε is the distance threshold and IoU is the intersection over union ratio. This not only takes into account the overlapping area between the prediction box and the label box, the center distance of the bounding box width, but also the scale characteristics of the aspect ratio of the bounding box, which makes the selected prediction box more accurate.

[0110] Classify the candidate boxes to get the class, and then get the detection image containing the class and the prediction box

[0111] Step S400: merge the first detection image and the second detection image, and select a target prediction frame from the first prediction frame and the second prediction frame to obtain a merged detection image.

[0112] Reference Fig. 9 , in the workpiece image reasoning stage, a first detection image containing a first prediction box is obtained; in the slice image reasoning stage, a second detection image containing a second prediction box is obtained. In the post-processing stage, high-quality candidate boxes play a vital role in target encirclement and subsequent classification and clustering. Usually, a method is used to merge the slice and overlapping candidate box results to remove duplicates and map them back to the original image, but this has defects because it is not rigorous to only use the size of the IOU threshold to evaluate the detection box. In this embodiment, precise and accurate error suppression is achieved based on the comprehensive candidate box area and the IOU threshold. The target prediction box is selected from the first prediction box and the second prediction box based on the numerical relationship between the area value of the first prediction box, the area value of the second prediction box and the prediction area threshold. Specifically, The area value of the detection box inferred by the network for each device each time is obtained through Then, we sort the bounding boxes based on the area. The threshold value and the set standard value Tb pass Compare and get the target prediction box Finally, precise and accurate error suppression is achieved by considering the comprehensive candidate box area and IOU threshold.

[0113] Reference Figure 7 , given an RGB image The backbone network generates feature maps f∈R C×H×W , the H and W of the feature map are Then the feature map f is passed through the CNN encoder to get the output embedding The CNN decoder is based on f enc and a certain number of learning object queries o∈R N×d As input, detection is performed through a feed-forward network, where d is the size of the encoder output embedding.

[0114] Reference Figure 6In step S500 of some embodiments, counting the combined detection image according to clustering to obtain the number of each type of artifacts includes the following steps:

[0115] Step S510, calculating the ratio of the distance between clusters and the distance within clusters based on the merged detection image to determine the number of target categories;

[0116] Step S520 , calculating the intra-class distance and the inter-class distance according to the number of target categories, and obtaining the number of each category of workpieces according to the intra-class distance and the inter-class distance.

[0117] After detecting the position of each workpiece on the input image using a category-independent detector, coordinate clipping is used to obtain each workpiece patch and extract the workpiece instance image with clear boundaries. Subsequently, each workpiece instance needs to be classified unsupervised. The comprehensive clustering prediction consists of two parts: clustering for predicting K and clustering for classification; the clustering for predicting K aims to determine the number of categories contained in the input set of all devices after cutting, while the clustering for classification uses the predicted K value to perform unsupervised clustering on the workpieces to obtain the final classification results of different categories.

[0118] PCA is used to reduce the dimension of the feature map to 96 while ensuring the maximum separation of each sample point in the new representation space. Since K-means cannot cluster two classes with the same mean (same cluster center point), the soft clustering Gaussian Mixture Model (GMM) can complete clustering by selecting components to maximize the posterior probability, and the posterior probability of each data point represents the possibility of belonging to each class, rather than judging that it completely belongs to a certain class. It has more accurate classification results than K-means, which is used to guide the number of quasi-classifications, that is, the number of target classes K. In evaluating the clustering results and measuring the clustering performance metric, Calinski-Harabasz-Score is used to calculate the ratio of the distance between class heaps and the distance within the cluster to finally determine the number of target classes K. Set the data set X with a capacity of N = {x0, x1, …x i The clusters are clustered into K classes, and the closeness within the class (intra-class distance) is measured by calculating the sum of the squares of the distances between each point in the class and the class center. The separation of the data set (inter-class distance) is measured by calculating the sum of the squares of the distances between the center points of each class and the center point of the data set. The calculation formula of the CH index is: Among them, B k is the covariance matrix between classes, W k is the covariance matrix of the intra-class data, c q represents the center point of class q, c e represents the center point of the data set, C qRepresents the number of data in class q, and represents the data set of class q. The larger the CH-Score, the better the classification quality. Subsequently, the K-means++ clustering method is used to assign classification labels to the determined number of target categories, adjust the size of the device instance to 192×192, and then use PCA to reduce the dimension to 32 to achieve the classification of multi-category devices.

[0119] In addition, during the training process of the workpiece classification model, the parameters of the workpiece classification model are adjusted through the position error loss function, confidence loss function and classification error loss function, so that the model can achieve the best balance when predicting the workpiece position, size and category.

[0120] For the position error loss function, the loss function needs to ensure that the anchor box or anchor point position predicted by the model for each feature point is as close to the actual position as possible, thereby improving the accuracy of target detection.

[0121] For the confidence loss function, for each grid cell, the model predicts the confidence of the object, that is, whether there is an object in the grid cell. The loss function needs to ensure that these confidence predictions are consistent with the actual situation, thereby improving the model's adaptability to different scenarios. If there is a target in the grid cell, the confidence loss will consider the possibility of the target's existence in this grid cell, that is, the confidence is Pr(Object). If there is no target in the grid cell, the confidence loss will not consider the possibility of the target's existence. Therefore, the confidence loss with object will consider the possibility of the target's existence, thereby more accurately measuring the model's prediction error for the target's position and shape. The confidence loss without object only considers whether there is a target in the grid cell, without considering other attributes of the target. The confidence loss function is: The first half of the formula calculates the confidence error when the object is present, while the second half of the formula calculates the confidence error when the object is not present. The value of the grid point that contains the object in the detection frame is 1, and the value of the grid point that does not contain the object is 0; The meaning of the above parameters is opposite, the grid point containing the object is 0, and the grid point not containing the object is 1; is the error between the predicted box value and the true box value.

[0122] For the classification error loss function, for each object in the grid cell, the model predicts the category to which it belongs. The loss function needs to ensure that these category predictions are as accurate as possible with the actual category, thereby improving the accuracy of target detection. The formula function for the classification error loss is:

[0123] In order to achieve accurate tuning and back propagation of the prediction box, when calculating the loss, the three parts of the loss are usually weighted and summed to obtain the final loss value. By continuously optimizing the loss function, the prediction results of the model can be closer to the true value, and the accuracy and precision of target detection can be improved. Among them, the classification loss uses cross entropy loss to solve the category prediction problem; the confidence loss uses mean square error loss to solve the problem of whether the target contains the prediction. By adjusting the hyperparameters α and β, the importance of these two parts of the loss can be balanced, thereby optimizing the overall performance of the model.

[0124] In a specific embodiment, verification is performed in a self-constructed data set. Because unsupervised clustering lacks label guidance, it may cause information loss or confusion, resulting in poor clustering performance. By predicting the optimal K value before clustering, data information can be effectively utilized and clustering accuracy can be improved. For this purpose, the comparative experiment of instance clustering mainly includes two aspects: the clustering accuracy under various combination strategies when K is predetermined and the prediction error of K under different combination strategies.

[0125] First, K is predicted by evaluating its quality in the range of 2 to 6. PCA reduces the dimension of image features to 96 when the image size is set to 128×128. Then GMM is used for label assignment. The clustering quality is subsequently evaluated to determine the best combination for predicting K, resulting in the lowest prediction error of 0.605. Therefore, in practical scenarios, the above configuration can be used to predict the number of device categories K in the target area. Subsequently, the size of the drawn device instance image is resized to 192×192, the dimension of the image features is reduced to 32 by PCA, and K-means++ clustering prediction K is used to achieve fine-grained device classification with a clustering accuracy of 91.11%. Compared with the specified true value K, this will lead to higher clustering accuracy, verifying that the final clustering accuracy of instance clustering achieved by combining the proposed optimal clustering accuracy and the optimal K prediction error can meet the classification clustering tasks for dense and complex targets in industrial scenarios.

[0126] After counting using the above workpiece counting method, there may be missed detection or false detection in extremely complex scenarios. The correction module can be used to manually add or delete existing target boxes to correct the number of devices. Specifically, when the workpiece counting method returns the counting result information, it will also return the specific coordinate information of each contour segmentation box to facilitate the user to correct the number of device statistics. The user can select one or more segmentation boxes on the terminal to delete, or manually add rectangular boxes to correct the number. In addition, the user can count the historical piece counting areas through the device cumulative difference module and the history viewing module.

[0127] An embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the training method of the workpiece counting model and the workpiece counting method as described above when executing the computer program.

[0128] The electronic device may be any intelligent terminal including a computer.

[0129] In general, for the hardware structure of the electronic device, the processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute related programs to implement the technical solutions provided in the embodiments of the present application.

[0130] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory can store an operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory, and the processor is called to execute the methods of the embodiments of this application.

[0131] The input / output interface is used to realize information input and output.

[0132] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.).

[0133] The bus transmits information between the various components of the device (such as the processor, memory, input / output interface, and communication interface). The processor, memory, input / output interface, and communication interface communicate with each other within the device through the bus.

[0134] An embodiment of the present application provides a computer storage medium storing computer executable instructions for executing the workpiece piece counting model training method and workpiece piece counting method as described above.

[0135] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the above description of this specification, the description of reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0136] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0137] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0141] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A workpiece counting method, characterized in that: include: Acquire a workpiece image, and input the workpiece image into a workpiece classification model; Cutting the workpiece image to obtain a slice image; Performing workpiece detection according to the workpiece image by a detector of the workpiece classification model to obtain a first detection image including a first prediction frame, and performing workpiece detection according to the slice image to obtain a second detection image including a second prediction frame; Merging the first detection image and the second detection image, and selecting a target prediction frame from the first prediction frame and the second prediction frame to obtain a merged detection image; The combined detection image is counted according to clustering to obtain the number of each type of workpiece.

2. The workpiece counting method according to claim 1, characterized in that: The step of cutting the workpiece image to obtain a slice image includes: The workpiece image is cut according to a preset step length to obtain a plurality of partially overlapping slice images.

3. The workpiece counting method according to claim 1, characterized in that: Performing workpiece detection by using a detector of the workpiece classification model includes: Perform feature extraction on the input image to obtain a first feature; Position-encode the first feature and map it to a low-dimensional space to obtain a coded feature; Decoding the encoded features to obtain decoded features; Perform detection according to the decoded features to obtain a detection image including a class and a prediction box; Among them, when the input image is the workpiece image, the prediction box is the first prediction box, and the detection image is the first detection image; when the input image is the slice image, the prediction box is the second prediction box, and the detection image is the second detection image.

4. The workpiece counting method according to claim 3, characterized in that: The decoding of the encoded features to obtain decoded features includes: Reshaping the encoded feature in width and height to obtain a reshaped feature; Upsampling the reshaped features to obtain upsampled features; Convolving a fusion result of the up-sampled feature and the encoded feature to obtain a convolution feature, and adding the convolution feature and the up-sampled feature to obtain a second feature; The second feature and the sum of the feature extraction results of the second feature are pooled to obtain a decoding feature.

5. The workpiece counting method according to claim 1, characterized in that: The detecting according to the decoded features to obtain a detection image including a class and a prediction box includes: Detecting according to the decoded features to obtain a candidate prediction box; Determine a first distance between a center of a bounding box of the workpiece image and the candidate prediction box; When the difference between the intersection-over-union ratio of the candidate prediction box and the bounding box and the first distance is less than a distance threshold, selecting the candidate prediction box as the prediction box; Classification is performed according to the candidate boxes to obtain classes, and then a detection image including the classes and the prediction boxes is obtained.

6. The workpiece counting method according to claim 5, characterized in that: The detecting according to the decoding feature to obtain a candidate prediction frame includes: Regressing the distance between the feature point of the decoded feature and the boundary of the workpiece image to obtain a bounding box; Allocating the bounding box to a corresponding prediction layer according to a workpiece scale in the bounding box, wherein the workpiece scale is determined according to a distance between a center point of the bounding box and a boundary of the workpiece image; Determine a positive sample area and a negative sample area according to the center point of the bounding box; Classifying the anchor frame as a positive sample or a negative sample according to the overlap between the anchor frame and the positive sample region and the overlap between the anchor frame and the negative sample region; Training the prediction layer according to the positive sample and the negative sample to obtain a training prediction box, and adjusting the parameters of the prediction layer according to the distance between the training prediction box and the center point of the bounding box to obtain a trained prediction layer; The candidate prediction box is obtained by detecting the decoded features through the trained prediction layer.

7. The workpiece counting method according to claim 1, characterized in that: The selecting a target prediction frame from the first prediction frame and the second prediction frame comprises: A target prediction box is selected from the first prediction box and the second prediction box according to a numerical relationship between an area value of the first prediction box, an area value of the second prediction box and a prediction area threshold.

8. The workpiece counting method according to claim 1, characterized in that: The step of counting the combined detection images according to clustering to obtain the number of each type of workpieces includes: The number of target categories is determined by calculating the ratio of the distance between clusters and the distance within the cluster according to the combined detection image; The intra-class distance and the inter-class distance are calculated according to the number of target categories, and the number of each category of workpieces is obtained according to the intra-class distance and the inter-class distance.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the workpiece counting method according to any one of claims 1 to 8 when executing the computer program.

10. A computer storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the workpiece counting method according to any one of claims 1 to 8.

Images