A vision-based stacked plate quantity recognition method and system

Abstract

The application discloses a kind of based on vision's stacked plate quantity identification method and system, belong to quantity identification technical field.The method includes: S1: build image acquisition unit, obtain the profile of the one side of the object group to be identified capable of showing quantity;S2: the profile obtained is preprocessed to data, to guarantee data quality;S3: by affine transformation and disturbance to image, realize data enhancement;S4: image data is artificially marked, then multi-scale pipeline neural network is used to extract and fuse features to data, and the prediction information corresponding to each layer plate is output;Similarity measure model is established according to the similarity of prediction information and real label, and loss is calculated;S5: the parameters of the output result are decoded, and the detection result is displayed.The application can solve the problem that manual reading quantity is difficult, easy visual fatigue, human resources waste is difficult to realize automation when stacked plate side is misaligned.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of quantity identification, and particularly relates to a vision-based stacked plate quantity identification method and system. BACKGROUND

[0002] Stacked structures are widely used in the mechanical, building material, non-ferrous metal and other industries, and the step of identifying the quantity of stacked plates is an essential step in the production and delivery process of products. Since the plates in the stacked structure are arranged in a non-aligned manner, it is difficult for workers to read the quantity, visual fatigue is easy to occur, human resources are wasted, and most work environments are extremely harsh. The low efficiency of the quantity reading step has become an important problem restricting the production and delivery speed of products. Therefore, it is urgent to develop a technology for automatically reading the quantity of stacked plates to improve the efficiency and accuracy of the plate quantity identification step. In the process of reading the quantity of stacked plates, computer deep learning vision detection technology can be used to improve the identification efficiency and accuracy and reduce labor costs, thereby greatly promoting the development of automation. SUMMARY

[0003] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0004] To this end, the present application aims to provide a vision-based stacked plate quantity identification method and system, which can solve the problems of difficulty in reading the quantity by workers, visual fatigue, and waste of human resources when the side surfaces of stacked plates are not aligned, and can realize automation.

[0005] To solve the above technical problems, the present application is implemented as follows:

[0006] The present application provides a vision-based stacked plate quantity identification method, which comprises the following steps:

[0007] S1: An image acquisition unit is built to obtain a contour map of a side of a group of objects to be identified that can display the quantity;

[0008] S2: Data preprocessing is performed on the obtained contour map to ensure data quality;

[0009] S3: Data enhancement is achieved by performing affine transformation and perturbation on the image;

[0010] S4: The image data is manually labeled, and then a multi-scale pipeline neural network is used to extract and fuse features of the data to output prediction information corresponding to each layer of plates. A similarity measurement model is established according to the similarity between the prediction information and the true label, and the loss is calculated;

[0011] S5: The output results are decoded and processed, and the detection results are displayed.

[0012] In addition, the vision-based stacked plate piece quantity identification method according to the present application can further have the following additional technical features.

[0013] In some embodiments, the image acquisition unit in step S1 is built by an industrial camera and a linear laser, the direction of the laser line is consistent with the stacking direction of the stacked plate pieces, the industrial camera is controlled to scan and take pictures along the stacking direction, and a sequence of images of the side surface of the stacked plate pieces is obtained.

[0014] In some embodiments, the preprocessing in step S2 includes first image data cleaning, image labeling, and second image data cleaning.

[0015] In some embodiments, the first cleaning of image data includes image similarity calculation on unlabeled images, screening out blurred and redundant images, and removing the same.

[0016] In some embodiments, the second cleaning of image data includes verification of the mapping relationship between the image and the label in the library, removal of invalid data, screening out data with missing labels or few labels, and relabeling.

[0017] In some embodiments, the image similarity calculation includes:

[0018] Mean square error (MSE) calculation: where m and n are the length and width of the two images I and K, and (i, j) represents the pixel value at the i-th row and j-th column in the image.

[0019] Peak signal-to-noise ratio (PSNR) calculation, where MAX I represents the maximum value of the color of the image point, and MSE is the mean square error of the image.

[0020] Structural similarity (SSIM) calculation, where X and Y represent an image, μ x and μ y represent the average values of X and Y, respectively, and represent the variances of X and Y, respectively, and δ xy represents the covariance of X and Y; C1=(K1L) 2 , C2=(K2L) 2 , and L is the number of image gray levels. For an 8-bit gray image, L=255, K1=0.01, and K2=0.03.

[0021] In some embodiments, the mathematical expression of the affine transformation in step S3 is:

[0022]

[0023] wherein, M represents an affine transformation matrix, s represents a scale factor, R represents a rotation matrix, t x and t y represents a translation matrix, and θ is a rotation angle.

[0024] In some embodiments, the content of the disturbance in step S3 includes randomly changing the brightness, contrast, saturation, and hue of the image to obtain a plurality of disturbance data.

[0025] In some embodiments, the content of the manual annotation of the image data in step S4 includes:

[0026] The received picture data is manually annotated for the stacking plate gap, and the coordinates x1, y1, x2, y2 of the starting and ending points of a line and the category cls are output and saved to the annotation file; the first list of the annotation file indicates whether it is a stacking plate gap cls, and the second, third, fourth, and fifth lists indicate the segmentation stacking plate gap coordinate values, a mapping relationship library between the annotated image and the label is established, and the data format is: [cls, x1 / w, y1 / h, x2 / w, y2 / h], wherein cls: 1 indicates that it is a stacking plate gap, and 0 indicates that it is not a stacking plate gap; w represents the image width, and h represents the image height;

[0027] The content of using a multi-scale pipeline neural network to extract and fuse features includes:

[0028] The input image is adjusted using bilinear interpolation scaling;

[0029] Three times of down-sampling are sequentially performed, each time the picture is scaled by a specific ratio, the channel number is adjusted, and then an activation process is performed; the channel numbers of the three down-samplings are increased by a factor of 2;

[0030] The results of the three times of down-sampling are fused: each scale result is up-sampled to the size of the previous scale, and then spliced after matching the channel numbers;

[0031] The loss calculated in step S4 is a total loss balancing multi-dimensional errors,

[0032]

[0033] wherein a, b, and c are weights, is a binary classification loss value on cls, ρ is the Euclidean distance between the predicted result and the real label stacking plate gap line segment center point, and |ν| is the predicted result and the real label stacking plate gap line segment angle similarity.

[0034] The embodiment of the present application also provides a vision-based stacked plate quantity identification system, which can realize the vision-based stacked plate quantity identification method in any one of the above; the system comprises:

[0035] An image acquisition unit configured to be capable of acquiring a contour map of a side of a group of objects to be identified that can display the quantity;

[0036] A data preprocessing module configured to be capable of performing data preprocessing on the acquired contour map to guarantee data quality;

[0037] A data enhancement module configured to be capable of realizing data enhancement by performing affine transformation and disturbance on the image;

[0038] A deep learning module configured to be capable of performing artificial labeling on the image data, and then adopting a multi-scale pipeline neural network to extract and fuse features of the data and output prediction information corresponding to each layer of plates; a similarity measurement model is established according to the similarity of the prediction information and real labels, and loss is calculated;

[0039] A display module configured to be capable of decoding and processing each parameter of the output result and displaying the detection result.

[0040] Compared with the prior art, the present application has at least the following beneficial effects:

[0041] In the embodiment of the present application, the vision-based stacked plate quantity identification method provided adopts an industrial camera + a linear laser to build an acquisition unit, and requires that the direction of the laser line is consistent with the stacking direction, so that the gap profile between the plates can be highlighted by the laser line, and the influence of interference such as plate surface texture and illumination on gap identification is reduced; the sequence images are spliced into a complete side contour map, so that the gap missing of a single image is avoided, and full-dimensional contour data is provided for subsequent detection;

[0042] In the embodiment of the present application, the vision-based stacked plate quantity identification method provided cleans the data twice (unlabeled images are screened for fuzzy and redundant images by using MSE / PSNR / SSIM; labeled images are verified for image-label mapping relationship to remove invalid data), so as to avoid low-quality and mislabeled data from misleading model training and improve data credibility; affine transformation (scaling, rotation, translation) + random disturbance (brightness, contrast, saturation, hue) is used to expand data diversity, improve the robustness and generalization ability of the model, and enable the model to still stably identify under different working environments (such as slight illumination change and slight plate inclination);

[0043] In the embodiment of the present application, the visual-based stacked plate quantity recognition method provided builds a multi-scale pipeline neural network, extracts different resolution features through three times of downsampling (channel number 64->128->256), and fuses through upsampling splicing, which can capture large-scale overall stacking structure and small-scale subtle gaps, avoiding quantity misjudgment caused by single scale missing details or global;

[0044] In the embodiment of the present application, the visual-based stacked plate quantity recognition method provided designs a weighted total loss of "two-classification loss (judging whether it is a gap) + Euclidean distance loss (correcting the position of the gap center point) + line segment angle similarity loss (correcting the direction of the gap line segment)", which can balance multi-dimensional errors and ensure the accuracy of the category judgment, position positioning and direction matching of the gap, thereby ensuring the accuracy of the plate quantity statistics.

[0045] In the embodiment of the present application, the visual-based stacked plate quantity recognition method provided realizes the direct association of "gap recognition-quantity conversion" through the labeling format of "[cls, x1 / w, y1 / h, x2 / w, y2 / h] (cls=1 / 0 distinguishes whether it is a gap, and the proportional coordinates eliminate the influence of image size)" and the subsequent decoding processing of "conf (gap prediction probability) + coordinate proportion", reducing the error transmission of intermediate links.

[0046] The visual-based stacked plate quantity recognition system of the present application can realize the visual-based stacked plate quantity recognition method, and thus at least has all the features and advantages of the visual-based stacked plate quantity recognition method, which will not be repeated here. Additional aspects and advantages of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The visual-based stacked plate quantity recognition method flow chart disclosed for an embodiment of the present application;

[0048] Figure 2 The image acquisition unit structure schematic diagram disclosed for an embodiment of the present application.

[0049] Explanation of reference signs:

[0050] 01-linear laser, 02-linear laser emitter, 03-mounting platform, 04-camera, 05-stacked plate. DETAILED DESCRIPTION

[0051] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0052] The embodiments of the present application will be described in detail below with reference to the drawings and specific examples and application scenarios.

[0053] Please refer to Figure 1 In some embodiments of the present application, a vision-based stacked plate quantity identification method is provided, including the following steps:

[0054] Step 1: Build an image acquisition unit to acquire the side profile data of the stacked plate, the laser line direction of the image acquisition unit is consistent with the stacking direction of the stacked plate, control the image acquisition unit to scan and take pictures along the stacking direction, acquire the sequence images of the side of the stacked plate, splice the above sequence images to obtain the side profile of the stacked plate, and send it into a deep learning model;

[0055] Step 2: Data preprocessing, used for image similarity calculation, cleaning of repeated images, and elimination of fuzzy and redundant image data;

[0056] Step 3: Data enhancement, mainly for affine transformation and disturbance of image data, to improve the robustness and generalization ability of the model;

[0057] Step 4: Deep learning, including building a multi-scale pipeline neural network for multi-scale image feature extraction and fusion, and further establishing a similarity measurement regression model according to the mapping relationship between the image acquisition module and the image and the label. Finally, according to the parameter meaning determined when the deep learning model is constructed, including [conf, x1 / w, y1 / h, x2 / w, y2 / h], conf corresponds to the prediction probability of the class cls, which is the gap between the stacked plates, the segmentation stacked plate gap proportion coordinate value [x1 / w, y1 / h, x2 / w, y2 / h], the parameters of the output result are decoded and processed, and the detection result is displayed.

[0058] In some embodiments of the present application, an image acquisition unit is built to obtain a complete side profile of the stacked plate, including:

[0059] An image acquisition unit is built by using an industrial camera and a linear laser, the direction of the laser line of the image acquisition unit is consistent with the stacking direction of the stacked plate, the image acquisition unit is controlled to scan and take pictures along the stacking direction, sequence images of the side surface of the stacked plate are acquired, the sequence images are spliced to obtain a side surface contour map of the stacked plate, and a complete side surface contour map of the stacked plate is obtained.

[0060] In some embodiments of the present application, data preprocessing is performed on the image data in the received image acquisition module, including twice cleaning of the image data:

[0061] Firstly, the image data is cleaned: image similarity calculation is performed on the unlabeled images, and the fuzzy and redundant images are screened out.

[0062] The image similarity calculation formula is: mean square error (MSE) calculation, wherein m and n are the length and width of the two images I and K, (i, j) represents the pixel value at the i-th row and j-th column in the image; peak signal-to-noise ratio (PSNR) calculation, wherein MAX I represents the maximum value of the color of the image point, if each sampling point is represented by 8 bits, it is 255. MSE is the mean square error of the image; structure similarity (SSIM) calculation, wherein X and Y represent two images respectively, μ x and μ y represent the average values of X and Y respectively, and represent the variances of X and Y respectively, δ xy represents the covariance of X and Y. C1=(K1L) 2 , C2=(K2L) 2 , L is the number of image gray levels, K1 and K2 are constant coefficients; for an 8-bit gray image, L=255, K1 is 0.01, and K2 is 0.03.

[0063] Secondly, the image data is cleaned: the mapping relationship between the images and the label is verified, invalid data is removed, data with missing labels and few labels is screened out, and re-labeling is performed.

[0064] In some embodiments of the present application, the data augmentation content of the labeled image data in step 3 includes:

[0065] Affine transformation: affine transformation is performed on the images in the existing data set to enhance the robustness and generalization ability of the model;

[0066] The mathematical expression of the affine transformation formula is: wherein,

[0067] M represents an affine transformation matrix, s represents a scale factor, R represents a rotation matrix, t x and t y represents a translation matrix, and θ is a rotation angle.

[0068] The disturbance in step 3 is: randomly changing the brightness, contrast, saturation and hue of the image. Here, the ColorJitter function in PyTorch is used to realize it, and the values of the above four attributes are all 0.5, indicating that each attribute value can be randomly changed in the range of (1-0.5)-(1+0.5).

[0069] Optionally, a deep learning multi-scale pipeline neural network is built for multi-scale image feature extraction, fusion, and further establishment of a similarity measurement regression model according to the mapping relationship between the image acquisition module and the image and the label.

[0070] In some embodiments of the present application, manual stacking plate gap labeling is performed on the received picture data, and the coordinates x1, y1, x2, y2 of the starting and ending points of the line, whether they are stacking plate gap categories cls, are output and saved to a.txt labeling file. The first list of the labeling file indicates whether it is a stacking plate gap [cls], and the second, third, fourth and fifth lists indicate the segmentation stacking plate gap coordinate values [x1 / w, y1 / h, x2 / w, y2 / h]. A mapping relationship library [cls, x1 / w, y1 / h, x2 / w, y2 / h] is established between the labeling image and the label. Its data format is [cls, x1 / w, y1 / h, x2 / w, y2 / h], wherein cls: 1 represents a stacking plate gap, 0 represents not a stacking plate gap, y1, y2, x1, x2 can represent the left endpoint [x1 / w, y1 / h] and the right endpoint [x2 / w, y2 / h] proportional coordinates respectively, and the stacking plate gap is labeled by a line segment determined by two points, the adjacent two plates are segmented, w represents the image width, and h represents the image height.

[0071] In some embodiments of the present application, a multi-scale network structure is proposed for feature extraction, and the input is an RGB image with a size of [3, 2448, 2048]. The input RGB image is first bilinearly interpolated to a size of [3, 640, 640], and then subjected to the first downsampling with a scale of 1 / 4 and a channel number of 64, followed by two times of CBR (convolution + batch normalization processing + Relu activation function). The second downsampling is performed with a scale of 1 / 4 and a channel number of 128, followed by two times of CBR (convolution + batch normalization processing + Relu activation function). The third downsampling is performed with a scale of 1 / 4 and a channel number of 256, followed by two times of CBR (convolution + batch normalization processing + Relu activation function). The results of each scale are respectively upsampled to the same size as the feature map of the previous scale, with the same channel number as the feature map of the previous scale, and are concatenated together (current channel number: 2*channel number of the feature map of the previous scale). The pipelines of the three scales are respectively subjected to regression.

[0072] In some embodiments of the present application, in each scale, an n*5*1-dimensional array is output, which respectively represents a layer of stacked plates [conf, x1 / w, y1 / h, x2 / w, y2 / h]; a similarity measurement model is established based on the similarity of the prediction result and the true label, and a loss function is calculated: a binary classification loss value on the prediction result cls is calculated, which is mathematically expressed as wherein y conf correspond to the prediction probability of the real label and the gap between the stacked plates, respectively; the Euclidean distance p of the prediction result and the center point of the gap line segment of the real label stacked plate is calculated, which is mathematically expressed as wherein (x c1 ,y c1 ) and (x c2 ,y c2 ) are the center coordinates of the prediction result and the gap line segment of the real label stacked plate, respectively; the angle similarity of the prediction result and the real label stacked plate gap line segment is calculated, which is mathematically expressed as wherein (x 11 ,y 11 ), (x 12 ,y 12 ), (x 21 ,y 21 ), (x 22 ,y 22 ) are the left end point start point and the right end point end point coordinates of the two prediction result and real label stacked plate gap line segments, respectively, and h and w represent the height and width of the image, respectively, and p and u are smaller, the more similar.

[0073] To balance the above three losses, the hyperparameters a, b, and c are weighted to sum the three losses as the total loss. The mathematical expression is:

[0074] Finally, according to the parameter meaning determined when the deep learning model is constructed, including [conf, x1 / w, y1 / h, x2 / w, y2 / h], conf corresponds to the prediction probability of the category cls being the stacked plate gap, the segmented stacked plate gap proportion coordinate value [x1 / w, y1 / h, x2 / w, y2 / h], the decoding processing is performed on each parameter of the output result, and the detection result is displayed. The decoding processing here can be understood as one-to-one correspondence between a column of the output result and a column of parameters in [conf, x1 / w, y1 / h, x2 / w, y2 / h].

[0075] Embodiment 1:

[0076] In this embodiment, a vision-based stacked plate quantity identification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0077] Figure 1 is a flowchart of a vision-based stacked plate quantity identification method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0078] Step S01, an image acquisition unit is built, which is used to acquire the side profile data of the stacked plate. The laser line direction of the image acquisition unit is consistent with the stacking direction of the stacked plate, the image acquisition unit is controlled to scan and take pictures along the stacking direction, the sequence images of the side of the stacked plate are acquired, the sequence images are spliced to obtain the side profile of the stacked plate, and the side profile is sent into the deep learning model;

[0079] Step S02, a data preprocessing module is used for image similarity calculation, cleaning of repeated images, and elimination of fuzzy and redundant image data;

[0080] Step S03, a data enhancement module is mainly used for affine transformation and disturbance of image data, to improve the robustness and generalization ability of the model;

[0081] Step S04, deep learning, a multi-scale pipeline neural network is built, which is used for multi-scale image feature extraction and fusion, and a similarity measurement regression model is further established according to the mapping relationship between the image acquisition module and the image and the label. Finally, the decoding processing is performed on each parameter of the output result according to the parameter meaning determined when the deep learning model is constructed;

[0082] Figure 2It is a structure schematic view of an image acquisition unit in a vision-based stacked plate quantity identification method according to an embodiment of the present application, comprising a line laser 01, a line laser emitter 02, a mounting platform 03 (movable up and down with servo), a camera 04 and a stacked plate 05.

[0083] In this embodiment, the image acquisition unit is built: for acquiring the side profile data of the stacked plate, the laser line direction of the image acquisition unit is consistent with the stacking direction of the stacked plate, the image acquisition unit is controlled to scan and take pictures along the stacking direction, the sequence images of the side of the stacked plate are obtained, the sequence images are spliced to obtain the side profile of the stacked plate, and the side profile is sent into a deep learning model.

[0084] The data preprocessing module is used for image similarity calculation, cleaning of repeated images, and elimination of blurred and redundant image data.

[0085] The first cleaning of image data: the image similarity calculation is performed on the unlabeled images, and the blurred and redundant images are eliminated.

[0086] The image similarity calculation formula: mean square error (MSE) calculation, wherein m and n are the length and width of the two images I and K, (i, j) represents the pixel value at the i-th row and j-th column in the image; peak signal-to-noise ratio (PSNR) calculation, wherein MAX I represents the maximum value of the color of the image point, if each sampling point is represented by 8 bits, it is 255. MSE is the mean square error of the image; structure similarity (SSIM) calculation, wherein X and Y represent two images, respectively, and mu x and mu y represent the average values of X and Y, respectively, and represent the variances of X and Y, respectively, and delta xy represents the covariance of X and Y. C1=(K1L) 2 , C2=(K2L) 2 , and L is the number of image gray levels, for an 8-bit gray image, L=255, K1 is 0.01, and K2 is 0.03.

[0087] The second cleaning of image data: the mapping relationship between the images and the labels is verified, invalid data is eliminated, data with missing labels and few labels is screened, and the data is labeled again.

[0088] In the embodiment of the present application, the data enhancement module: mainly performs affine transformation and disturbance on the image data, to improve the robustness and generalization ability of the model.

[0089] The mathematical expression of the affine transformation formula is: wherein, M represents an affine transformation matrix, s represents a scale factor, R represents a rotation matrix, t x and t y represents a translation matrix, and θ is a rotation angle.

[0090] Disturbance: randomly change the brightness, contrast, saturation, and hue of the image. Here, the ColorJitter function in PyTorch is used to implement this, and the values of the above four attributes are all 0.5, indicating that each attribute value can be randomly changed in the range of (1-0.5)-(1+0.5).

[0091] In this embodiment, deep learning is used to build a multi-scale pipeline neural network for multi-scale image feature extraction, fusion, and further establishment of a similarity measurement regression model according to the mapping relationship between the image acquisition module and the image and the label.

[0092] In this embodiment, manual stacking plate gap labeling is performed on the received picture data, and the coordinates x1, y1, x2, y2 of the starting and ending points of the line, whether it is a stacking plate gap class cls, are output and saved to a.txt labeling file. The first list of the labeling file indicates whether it is a stacking plate gap [cls], and the second, third, fourth, and fifth lists indicate the segmentation stacking plate gap coordinate values [x1 / w, y1 / h, x2 / w, y2 / h]. A mapping relationship library [cls, x1 / w, y1 / h, x2 / w, y2 / h] is established between the labeling image and the label. Its data format is [cls, x1 / w, y1 / h, x2 / w, y2 / h], wherein cls: 1 indicates that it is a stacking plate gap, and 0 indicates that it is not a stacking plate gap. y1, y2, x1, and x2 can represent the left endpoint [x1 / w, y1 / h] and the right endpoint [x2 / w, y2 / h] proportional coordinates, respectively. The stacking plate gap is labeled by a line segment determined by two points, and the adjacent two plates are segmented. w represents the image width, and h represents the image height.

[0093] The input is an RGB image with a size of [3, 2448, 2048], a multi-scale network structure is proposed for feature extraction. First, the original RGB image is bilinearly interpolated to a size of [3, 640, 640], the first down-sampling is performed with a scale of 1 / 4 and a channel number of 64, followed by two times of CBR (convolution + batch normalization processing + Relu activation function), the second down-sampling is performed with a scale of 1 / 4 and a channel number of 128, followed by two times of CBR (convolution + batch normalization processing + Relu activation function), the third down-sampling is performed with a scale of 1 / 4 and a channel number of 256, followed by two times of CBR (convolution + batch normalization processing + Relu activation function). Each scale result is up-sampled to the same size as the feature map of the previous scale, with the same number of channels as the feature map of the previous scale, and is concatenated together (current channel number: 2*previous scale feature map channel number), and the pipeline of the three scales is regressed respectively.

[0094] In each scale, an n*5*1-dimensional array is output, which represents a layer of plates [conf, x1 / w, y1 / h, x2 / w, y2 / h] respectively; a similarity measure model is established based on the similarity of the prediction result and the true label, and a loss function is calculated: a binary classification loss value on the prediction result cls is calculated, which is mathematically expressed as where y conf correspond to the prediction probability of the real label and the gap between the stacked plates respectively; the Euclidean distance p between the center points of the prediction result and the real label gap line segment is calculated, which is mathematically expressed as where (x c1 ,y c1 ) and (x c2 ,y c2 ) are the center coordinates of the prediction result and the real label gap line segment respectively; the angle similarity between the prediction result and the real label gap line segment is calculated, which is mathematically expressed as where (x 11 ,y 11 ), (x 12 ,y 12 ), (x 21 ,y 21 ), (x 22 ,y 22 ) are the left end point start point and right end point end point coordinates of the prediction result and the real label two stacked plate gap line segments respectively, h and w represent the height and width of the image respectively, and p and u are smaller, the more similar.

[0095] To balance the above three losses, the hyperparameters a, b, and c are weighted to sum the three losses as the total loss. The mathematical expression is:

[0096] Finally, according to the parameter meanings determined when the deep learning model is constructed, including [conf, x1 / w, y1 / h, x2 / w, y2 / h], conf corresponds to the prediction probability of the category cls being the stacked plate gap, the segmented stacked plate gap proportion coordinate values [x1 / w, y1 / h, x2 / w, y2 / h], the parameters of the output result are decoded and processed, and the detection result is displayed.

[0097] The parts not described in detail in the present application can refer to the prior art or be known to those skilled in the art, and the present embodiment is not limited thereto, and will not be described in detail here.

[0098] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative and not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A vision-based method for recognizing the number of stacked boards, characterized in that, The method includes: S1: Set up an image acquisition unit to obtain the outline of the number of sides of the object group to be identified that can be displayed; S2: Perform data preprocessing on the acquired contour map to ensure data quality; S3: Data augmentation is achieved by applying affine transformations and perturbations to the image; S4: Manually label the image data, then use a multi-scale pipeline neural network to extract and fuse features from the data, and output the prediction information corresponding to each layer of the board; establish a similarity measurement model based on the similarity between the prediction information and the real labels, and calculate the loss. The content of using multi-scale pipeline neural networks for feature extraction and fusion of data includes: The input image is adjusted using bilinear interpolation scaling; The image is downsampled three times in sequence, and each time the image is scaled by a specific ratio and the number of channels is adjusted, followed by activation processing; the number of channels in the three downsamples increases by a factor of 2. The three downsampling results are fused: the results at each scale are upsampled to the size of the previous scale, and the channels are matched before being stitched together. S5: Decode the parameters of the output results and display the detection results.

2. The vision-based method for recognizing the number of stacked boards according to claim 1, characterized in that, In step S1, an image acquisition unit is built using an industrial camera and a linear laser. The laser line direction is consistent with the stacking direction of the stacked boards. The industrial camera is controlled to scan and take pictures along the stacking direction to obtain a sequence of images of the side of the stacked boards. The above sequence of images are stitched together to obtain a complete side profile of the stacked boards.

3. The vision-based method for recognizing the number of stacked boards according to claim 1, characterized in that, The preprocessing in step S2 includes first image data cleaning, image annotation, and second image data cleaning.

4. The vision-based method for recognizing the number of stacked boards according to claim 3, characterized in that, The first step of cleaning image data involves calculating the image similarity of unlabeled images, filtering out blurry and redundant images, and removing them.

5. The vision-based method for recognizing the number of stacked boards according to claim 3, characterized in that, The second step involves cleaning the image data: verifying the mapping relationship between the labeled images and the label database, removing invalid data, filtering out data with missing or incomplete labels, and then labeling them again.

6. The vision-based method for recognizing the number of stacked boards according to claim 4, characterized in that, The image similarity calculation includes: Mean Square Error (MSE) Calculation: , where m and n are the length and width of images I and K, and (i, j) represents the pixel value at the i-th row and j-th column of the image; Peak signal-to-noise ratio (PSNR) calculation ,in, This represents the maximum value of the color of a point in the image. The mean square error of the image; Structural similarity (SSIM) calculation Where X and Y each represent an image, and Let X and Y represent the average values, respectively. and Let X and Y represent the variances, respectively. This represents the covariance of X and Y; , , L The number of gray levels in the image. , These are constant coefficients.

7. The vision-based method for recognizing the number of stacked boards according to claim 1, characterized in that, The mathematical expression for the affine transformation in step S3 is: ， in, , M represents the affine transformation matrix, s represents the scale factor, and R represents the rotation matrix. and Represents the translation matrix. The angle is the rotation angle.

8. The vision-based method for recognizing the number of stacked boards according to claim 1, characterized in that, The perturbation in step S3 includes randomly changing the four values ​​of brightness, contrast, saturation, and hue of the image to obtain perturbed data.

9. The vision-based method for recognizing the number of stacked boards according to claim 1, characterized in that, The manual annotation of image data in step S4 includes: The received image data is manually annotated with the gaps between stacked panels. The coordinates of the start and end points of a line (x1, y1, x2, y2) and the category (cls) are output and saved to the annotation file. The first column of the annotation file indicates whether it is a gap between stacked panels (cls), and the second, third, fourth, and fifth columns represent the coordinate values ​​of the gaps between stacked panels. A mapping relationship library between the annotated image and the label is established, with the data format: [cls, x1 / w, y1 / h, x2 / w, y2 / h], where cls: 1 indicates a gap between stacked panels, and 0 indicates no gap between stacked panels; w represents the image width, and h represents the image height. The loss calculated in step S4 is the total loss that balances the multidimensional errors. ， Where a, b, and c are all weights. The binary classification loss value on cls, The predicted distance is the Euclidean distance between the center point of the gap segment of the actual label stacked boards. The similarity between the predicted result and the included angle of the gap line segment of the actual label stacked board is used.

10. A vision-based system for recognizing the number of stacked boards, characterized in that, The system is capable of implementing the vision-based stacked board quantity recognition method as described in any one of claims 1 to 9; the system comprises: The image acquisition unit is configured to acquire a contour map of the number of sides of the group of objects to be identified that can be displayed; The data preprocessing module is configured to preprocess the acquired contour map to ensure data quality. The data augmentation module is configured to perform data augmentation by applying affine transformations and perturbations to the image. The deep learning module is configured to manually annotate image data, then use a multi-scale pipeline neural network to extract and fuse features from the data, and output prediction information for each layer of the board; a similarity measurement model is established based on the similarity between the prediction information and the real labels, and the loss is calculated. The display module is configured to decode the parameters of the output results and display the detection results. The content of using multi-scale pipeline neural networks for feature extraction and fusion of data includes: The input image is adjusted using bilinear interpolation scaling; The image is downsampled three times in sequence, and each time the image is scaled by a specific ratio and the number of channels is adjusted, followed by activation processing; the number of channels in the three downsamples increases by a factor of 2. The three downsampling results are fused: the results at each scale are upsampled to the size of the previous scale, and the channels are matched before being stitched together.

Images