A method and system for accurate counting of dense copper sheets based on semantic segmentation and image processing

Abstract

This invention discloses a method and system for accurate counting of dense copper sheets based on semantic segmentation and image processing. The method includes: acquiring a large field-of-view image of dense copper sheets; labeling and cropping the copper sheets in the image, and inputting the cropped image and labels into a Unet semantic segmentation network for training to obtain a trained Unet network model; setting the input image of the trained Unet network model to the original image size, and performing inference on the original image to obtain a preliminary inference map; performing an opening operation on the preliminary inference map to filter out small imperfections, and performing dilation to obtain large columns of copper sheets; correcting the angle of the minimum bounding rectangle of the large columns of copper sheets; and forming a peak-and-valley image through affine transformation, projection integration, and filtering, and counting the number of copper sheets in all large columns of copper sheets in the image by counting the number of peaks. This invention effectively solves the problem of achieving high-precision counting in scenes with mottled and densely arranged copper sheets.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and relates to the accurate identification and counting of specific objects that are closely arranged and stacked, and in particular to a method for accurate segmentation and counting of high-resolution copper sheet images based on semantic segmentation and image processing. Background Technology

[0002] With the rapid development of computer vision technology, cameras are increasingly used in various fields such as real-time intelligent monitoring, traffic control, industrial inspection, and medical imaging. In the field of industrial inspection, cameras are often used to identify and count objects instead of manual labor, saving a significant amount of manpower and time while also providing greater safety. However, current counting algorithms struggle to achieve high-precision counting for densely packed copper sheets with mottled surfaces. Therefore, researching a counting algorithm specifically for densely packed copper sheet targets is of great significance.

[0003] Currently, methods for identification and counting can generally be divided into traditional methods, object detection, semantic segmentation, etc.

[0004] (1) Traditional detection algorithms usually use feature extraction and line detection, contour detection and other methods. Although they are simple to calculate, they are not robust and require a lot of parameters to be adjusted. They are only suitable for simple scenarios.

[0005] (2) Deep learning-based target detection algorithms mainly identify and record targets. The target detection algorithm will directly output the number of targets, which is suitable for real-time counting of sparse targets in dynamic scenes. However, the false detection rate is high for high similarity scenes such as copper sheets.

[0006] (3) Semantic segmentation algorithms classify each pixel, and the same category in the inference result image is assigned the same color (usually a black and white image when there is only one target, representing the background area and the target respectively). During inference, the network does not directly output the number of targets, so it is often combined with post-processing algorithms for target counting and accuracy optimization. The commonly used method is to perform contour detection and determine the number of targets based on the number of contours. However, due to the high similarity of copper sheet scenes, there may be some targets sticking together or splitting in the semantic segmentation result. It is difficult to accurately count them using contour detection. The commonly used method is to filter by contour area or length, but it is difficult to choose the filtering threshold. Another commonly used method is to use erosion to break the connection of sticking, but it is difficult to choose the shape and size of the erosion structure element for different situations. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention employs a semantic segmentation algorithm for accurate counting of dense copper sheets. By training the Unet semantic segmentation model through image cropping, the accuracy of copper sheet recognition is improved. Simultaneously, this invention proposes an affine transformation and pixel projection integration method, transforming the counting problem into a problem of counting peaks and troughs, thus resolving the issues of target adhesion and discontinuity in the initial segmentation image, and is unaffected by the arrangement direction of the copper sheets in the image. The main purpose of this invention is to provide a method and system for accurate counting of dense copper sheets based on semantic segmentation and image processing, overcoming the aforementioned problems in the prior art.

[0008] To achieve the aforementioned objectives, the present invention employs the following solution:

[0009] One aspect of the present invention provides a method for accurate counting of dense copper sheets based on semantic segmentation and image processing, comprising the following steps:

[0010] Step 1: Use the low magnification mode of the gimbal camera to acquire images of dense copper sheets with a large field of view;

[0011] Step 2: Label the copper pieces in the copper sheet image, crop the labeled mask and the original image, and input the cropped image and label into the Unet semantic segmentation network for training to obtain the trained Unet network model;

[0012] Step 3: Set the input image to the original image size, and use the trained network model to infer the original image to obtain a preliminary inference graph;

[0013] Step 4: Binarize the preliminary inference graph, then remove the small black blocks by opening operation, and then expand the preliminary inference graph after removing the small stains to combine the copper pieces in the same column to form a large copper piece column, generating a large copper piece column set.

[0014] Step 5: Select the first large column of copper sheets from the set of large columns of copper sheets, obtain the minimum bounding rectangle of the first large column of copper sheets, and correct the minimum bounding rectangle of the first large column of copper sheets.

[0015] Step 6: Affine transform the smallest bounding rectangle region of the first copper sheet column after correction to another image to generate a columnar copper sheet image; stack the black pixels representing the copper sheets on the horizontal axis by pixel projection to form a peak and valley image; process the peak and valley image to obtain the number of peaks, which is the number of copper sheets in the first copper sheet column.

[0016] Step 7: Repeat steps 5 and 6 until all the copper sheet columns in the copper sheet column set have been counted, and count the number of copper sheets in all the copper sheet columns in the image.

[0017] In one embodiment, the process of correcting the minimum bounding rectangle of the first copper sheet column in step five includes: obtaining the minimum bounding rectangle of all copper sheets in the first copper sheet column, calculating the average angle of the minimum bounding rectangle of all copper sheets in the first copper sheet column, and using the average angle to correct the angle of the minimum bounding rectangle of the first copper sheet column, so that it is rotated to an angle parallel or perpendicular to the copper sheet.

[0018] In one embodiment, step six, based on obtaining the number of peaks, includes: filtering the peak and trough image to eliminate redundant spikes on the peaks; and determining the peaks using the maximum point method to obtain the number of peaks.

[0019] In one embodiment, the affine transformation process in step six includes: obtaining the coordinates of the four corners of the minimum bounding rectangle, and then obtaining the coordinates of the four corners of the other image; obtaining four pairs of matching points based on the coordinates of the four corners of the minimum bounding rectangle and the coordinates of the four corners of the other image, calculating the affine transformation matrix using the least squares method, and then performing the affine transformation using the affine transformation matrix.

[0020] In one embodiment, each column of the peak and trough image has a certain number of black pixels. It is determined whether the number of pixels in each column is a maximum point. A maximum point is a point that achieves the maximum value within a certain neighborhood on the left and right.

[0021] Another aspect of the present invention provides a dense copper sheet precision counting system based on semantic segmentation and image processing, the counting system comprising:

[0022] An image acquisition module is used to acquire images of dense copper sheets with a large field of view using the low magnification mode of a gimbal camera.

[0023] The annotation and cropping module is used to annotate the copper pieces in the copper piece image, crop the annotated mask labels and the original image, and input the cropped image and labels into the Unet semantic segmentation network for training to obtain the trained Unet network model.

[0024] The preliminary inference module is used to set the input image to the original image size and to obtain a preliminary inference image by inferring from the original image through the trained network model.

[0025] The inference graph processing module is used to perform binarization processing on the preliminary inference graph, then remove small black blocks through opening operation, and then perform expansion operation on the preliminary inference graph after removing small stains to combine copper pieces in the same column to form a large copper piece column, generating a large copper piece column set.

[0026] A copper sheet column correction module is used to select a first copper sheet column from the set of copper sheet columns, obtain the minimum bounding rectangle of the first copper sheet column, and correct the minimum bounding rectangle of the first copper sheet column.

[0027] The statistical counting module is used to perform an affine transformation on the smallest bounding rectangle region of the corrected first copper sheet column to another image, generating a columnar copper sheet image; the black pixels representing the copper sheets are stacked on the horizontal axis by pixel projection to form a peak-valley image; the peak-valley image is processed to obtain the number of peaks, which is the number of copper sheets in the first copper sheet column.

[0028] In one embodiment, the copper sheet column correction module is further configured to: obtain the minimum bounding rectangle of all copper sheets in the first copper sheet column, calculate the average value of the angles of the minimum bounding rectangles of all copper sheets in the first copper sheet column, and use the average value of the angles to correct the angles of the minimum bounding rectangles of the first copper sheet column, so that they are rotated to an angle parallel or perpendicular to the copper sheets.

[0029] In one embodiment, the statistical counting module is further configured to: filter the peak and trough image to eliminate redundant spikes on the peaks; and determine the peaks by using the maximum point method to obtain the number of peaks.

[0030] In one embodiment, the statistical cardinality module is further configured to: obtain the coordinates of the four corners of the minimum bounding rectangle, and then obtain the coordinates of the four corners of the other image; obtain four pairs of matching points based on the coordinates of the four corners of the minimum bounding rectangle and the coordinates of the four corners of the other image, calculate the affine transformation matrix using the least squares method, and then perform an affine transformation using the affine transformation matrix.

[0031] In one embodiment, each column of the peak and trough image has a certain number of black pixels. It is determined whether the number of pixels in each column is a maximum point. A maximum point is a point that achieves the maximum value within a certain neighborhood on the left and right.

[0032] Compared with the prior art, the present invention has at least the following advantages: (1) The present invention uses a semantic segmentation algorithm to accurately count dense copper pieces. By cropping the image, the Unet semantic segmentation model is trained, which improves the accuracy of copper piece recognition; (2) The present invention proposes an affine transformation and pixel projection integration method to transform the counting problem into the problem of counting peaks and valleys, which solves the problem of target adhesion and discontinuity in the preliminary segmentation image, and is not affected by the arrangement direction of the copper pieces in the image. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a hardware architecture diagram of a dense copper sheet precision counting system based on semantic segmentation and image processing according to one embodiment of the present invention;

[0035] Figure 2 This is the original image of the copper sheet in one embodiment of the present invention;

[0036] Figure 3 This is a preliminary sectioning diagram of the copper sheet according to one embodiment of the present invention;

[0037] Figure 4 It is the largest column of copper sheets and the smallest bounding rectangle after the angle correction in one embodiment of the present invention;

[0038] Figure 5 This is a projection integral peak and valley diagram and a peak comparison diagram before and after the filtering algorithm in one embodiment of the present invention;

[0039] Figure 6 This is a flowchart of a method for accurate counting of dense copper sheets based on semantic segmentation and image processing according to an embodiment of the present invention. Detailed Implementation

[0040] In view of the shortcomings of the prior art, the inventors have proposed the technical solution of the present invention, which effectively solves the problem of difficulty in achieving high-precision counting in the case of copper sheets with mottled surfaces and dense arrangement, and can achieve efficient and accurate counting.

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Examples of these preferred embodiments are illustrated in the drawings. The embodiments of the present invention shown in and described with reference to the drawings are merely exemplary, and the present invention is not limited to these embodiments.

[0042] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0043] In a typical embodiment of the present invention, the technical solution is implemented as follows:

[0044] 1) The entire system mainly includes a PTZ camera, network cable, host (3080PI graphics card), and monitor.

[0045] 2) Use the low magnification mode of the gimbal camera to acquire images of copper sheets with a large field of view, use annotation software to annotate the copper sheets, crop the annotated mask labels and the original images, and put the cropped images and labels into the Unet semantic segmentation network for training to obtain the network model.

[0046] 3) Set the input image to the original image size. Perform inference on the original image to obtain a preliminary inference map. In the preliminary inference map, the copper pieces are basically segmented, but there are many noise black spots and some truncation and adhesion. Here, the original image refers to the image without cropping; inference refers to using a trained deep learning network model to infer the untrained image, assigning a category to each pixel of the image (including background and copper piece categories), and assigning different colors to different categories.

[0047] 4) The Otsu algorithm was used for preliminary reasoning. Figure 2 The data is valued and then small black blocks are removed using an opening operation to reduce interference with subsequent processes. A larger expansion operation is then performed on the initial inference graph to combine copper sheets in the same column into a larger column.

[0048] 5) Obtain the minimum bounding rectangle of a large column of copper sheets using the OpenCV function minAreaRect(). Then, obtain the minimum bounding rectangles of all copper sheets in this column and calculate the average angle of all minimum bounding rectangles in this column. Use the average angle to correct the angle of the minimum bounding rectangle of the large column of copper sheets, rotating it to an angle parallel or perpendicular to the copper sheets.

[0049] 6) Affine transform the smallest bounding rectangle region of the large copper sheet column to another image to generate a histogram-like image of the copper sheet column. Pixels are stacked on the x-axis using pixel projection to form a peak-and-trough image. The image is then filtered to eliminate unnecessary spikes on the peaks. The peaks are determined by identifying extreme points, and the number of peaks represents the number of copper sheets in that large copper sheet column.

[0050] Each column of the peak and trough image has a certain number of black pixels. The goal is to determine whether the number of black pixels in each column is a maximum point. A maximum point is a point that achieves the maximum value within a certain neighborhood to the left and right. The determination method is as follows: if the number of black pixels in the current column is greater than that in the previous column and greater than that in the next column, or if the number of black pixels in the current column is greater than that in the previous column and equal to that in the next column (record the number of black pixels in the current column, and continue to check if the next column is equal. If they are equal, continue. If they are not equal, determine whether the number of black pixels is greater than or less than the recorded number of black pixels. If it is less than, it is classified as a maximum point; otherwise, it is not). The above two cases are maximum points, and other cases are not maximum points.

[0051] The other image here is an empty image with a fixed size. Its width is approximately the same as the height of the original image, and its height is approximately the same as the width of the copper sheet.

[0052] The angle of the bounding rectangle refers to the angle between the bounding rectangle and the horizontal axis. The minimum bounding rectangle of the large column of copper sheets may not be perpendicular to the copper sheets, causing the copper sheets to be tilted after the affine transformation, which is detrimental to counting. Therefore, it is necessary to pre-correct the angle of the minimum bounding rectangle of the large column of copper sheets to make it perpendicular to the copper sheets. The correction process is as follows: calculate the minimum bounding rectangle and angle of each copper sheet, and set the angle of the minimum bounding rectangle of the large column of copper sheets to the average of the angles of the minimum bounding rectangles of all copper sheets.

[0053] The specific process of reflection transformation is as follows: obtain the coordinates of the four corners of the smallest bounding rectangle, then obtain the coordinates of the four corners of the other image mentioned above, thus obtaining four pairs of matching points. Calculate an affine transformation matrix using the least squares method, and then use this transformation matrix to perform an affine transformation.

[0054] 7) Repeat steps 5) and 6) until all copper strip columns have been counted, and finally count the number of copper strips in all copper strip columns in the image.

[0055] Specifically, such as Figure 1 As shown, a fixed support pole is used to mount the sphere camera above the copper sheet scene. After the camera acquires image data, it is transmitted to the host computer for analysis and processing. The specific implementation steps are as follows:

[0056] 1) such as Figure 2 The image shown is a high-resolution image of a copper sheet captured by a PTZ camera. It can be seen that the scene colors are quite similar, the features of the target are not obvious, and the copper sheets are arranged in a dense manner, with some areas showing misalignment.

[0057] 2) Label and cut the copper sheets, then feed them into the network for training, such as... Figure 3As shown, the segmentation image obtained from the initial inference using the trained Unet network consists of black and white colors, with the black areas representing the copper sheet target. It can be seen that the initially segmented image contains breaks, adhesions, and small blemishes.

[0058] 3) such as Figure 4 As shown, a large column of copper sheets is obtained through significant expansion, and the angle of the minimum bounding rectangle of the large column of copper sheets is corrected by the average angle of the minimum bounding rectangle of all copper sheets in the current column. Figure 4 The right-hand image shows the smallest bounding rectangle of the large column of copper sheets after correction, which is perpendicular to the copper sheets.

[0059] 4) such as Figure 5 As shown, the affine transformation of the outer rectangular region of the large column of copper sheet is applied to another image, and then the peak and valley map is obtained by projection integration. After filtering, it can be seen that there is only one true peak after filtering. Figure 5 (a) in the image is the peak diagram before filtering. Figure 5 (b) in the image is the filtered peak diagram.

[0060] 5) The entire counting method process is as follows: Figure 6 As shown, the process involves first acquiring images, performing initial segmentation using the U-net semantic segmentation algorithm, then filtering out small blemishes using open operations, obtaining the minimum bounding rectangle of the large column of copper sheets perpendicular to the copper sheets, obtaining peak and valley images using affine transformation and filtering them, and finally counting the results.

[0061] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions of each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for accurate counting of dense copper sheets based on semantic segmentation and image processing, characterized in that... Includes the following steps: Step 1: Use the low magnification mode of the gimbal camera to acquire images of dense copper sheets with a large field of view; Step 2: Label the copper pieces in the copper sheet image, crop the labeled mask and the original image, and input the cropped image and label into the Unet semantic segmentation network for training to obtain the trained Unet network model; Step 3: Set the input image to the original image size, and use the trained network model to infer the original image to obtain a preliminary inference graph; Step 4: Binarize the preliminary inference graph, then remove the small black blocks by opening operation, and then expand the preliminary inference graph after removing the small stains to combine the copper pieces in the same column to form a large copper piece column, generating a large copper piece column set. Step 5: Select the first large column of copper sheets from the set of large columns of copper sheets, obtain the minimum bounding rectangle of the first large column of copper sheets, and correct the minimum bounding rectangle of the first large column of copper sheets. Step 6: Affine transform the smallest bounding rectangle region of the first large column of copper sheets after correction to another image to generate a columnar image of copper sheets. By using pixel projection, black pixels representing copper sheets are stacked on the horizontal axis to form a peak-and-valley image; The peak and trough image is processed to obtain the number of peaks, which is the number of copper sheets in the first copper sheet column. Step 7: Repeat steps 5 and 6 until all the copper sheet columns in the copper sheet column set have been counted, and count the number of copper sheets in all the copper sheet columns in the image.

2. The counting method according to claim 1, characterized in that, The process of correcting the minimum bounding rectangle of the first copper sheet column in step five includes: obtaining the minimum bounding rectangle of all copper sheets in the first copper sheet column, calculating the average angle of the minimum bounding rectangle of all copper sheets in the first copper sheet column, and using the average angle to correct the angle of the minimum bounding rectangle of the first copper sheet column, so that it is rotated to an angle parallel or perpendicular to the copper sheet.

3. The counting method according to claim 1, characterized in that, The process of obtaining the number of peaks in step six includes: filtering the peak and trough image to eliminate unnecessary spikes on the peaks; and using the maximum point method to determine the peaks and obtain the number of peaks.

4. The counting method according to claim 2, characterized in that, The affine transformation process in step six includes: obtaining the coordinates of the four corners of the minimum bounding rectangle, and then obtaining the coordinates of the four corners of the other image; obtaining four pairs of matching points based on the coordinates of the four corners of the minimum bounding rectangle and the coordinates of the four corners of the other image, calculating the affine transformation matrix using the least squares method, and then performing the affine transformation using the affine transformation matrix.

5. The counting method according to claim 3, characterized in that, Each column of the peak and trough image has a certain number of black pixels. The goal is to determine whether the number of black pixels in each column is a maximum point. A maximum point is a point that achieves the maximum value within a certain neighborhood to the left and right.

6. A dense copper sheet precision counting system based on semantic segmentation and image processing, characterized in that, The counting system includes: An image acquisition module is used to acquire images of dense copper sheets with a large field of view using the low magnification mode of a gimbal camera. The annotation and cropping module is used to annotate the copper pieces in the copper piece image, crop the annotated mask labels and the original image, and input the cropped image and labels into the Unet semantic segmentation network for training to obtain the trained Unet network model. The preliminary inference module is used to set the input image to the original image size and to obtain a preliminary inference image by inferring from the original image through the trained network model. The inference graph processing module is used to perform binarization processing on the preliminary inference graph, then remove small black blocks through opening operation, and then perform expansion operation on the preliminary inference graph after removing small stains to combine copper pieces in the same column to form a large copper piece column, generating a large copper piece column set. A copper sheet column correction module is used to select a first copper sheet column from the set of copper sheet columns, obtain the minimum bounding rectangle of the first copper sheet column, and correct the minimum bounding rectangle of the first copper sheet column. The statistical counting module is used to perform an affine transformation on the smallest bounding rectangle region of the corrected first copper sheet column to another image, generating a columnar copper sheet image; the black pixels representing the copper sheets are stacked on the horizontal axis by pixel projection to form a peak-valley image; the peak-valley image is processed to obtain the number of peaks, which is the number of copper sheets in the first copper sheet column.

7. The counting system according to claim 6, characterized in that, The copper sheet column correction module is further configured to: obtain the minimum bounding rectangle of all copper sheets in the first copper sheet column, calculate the average angle of the minimum bounding rectangle of all copper sheets in the first copper sheet column, and use the average angle to correct the angle of the minimum bounding rectangle of the first copper sheet column, so that it is rotated to an angle parallel or perpendicular to the copper sheet.

8. The counting system according to claim 6, characterized in that, The statistical counting module is also used to: filter the peak and trough image to eliminate unnecessary spikes on the peaks; and determine the peaks by using the maximum point method to obtain the number of peaks.

9. The counting system according to claim 7, characterized in that, The statistical counting module is also used to: obtain the coordinates of the four corners of the minimum bounding rectangle, and then obtain the coordinates of the four corners of the other image; obtain four pairs of matching points based on the coordinates of the four corners of the minimum bounding rectangle and the coordinates of the four corners of the other image, calculate the affine transformation matrix using the least squares method, and then perform an affine transformation using the affine transformation matrix.

10. The counting system according to claim 8, characterized in that, Each column of the peak and trough image has a certain number of black pixels. The determination is made as to whether the number of pixels in each column is a maximum point. A maximum point is a point that achieves the maximum value within a certain neighborhood on the left and right.

Images