A vision-based method and system for counting stacks of steel plates

Abstract

The application discloses a kind of steel plate stacking inventory method and system based on vision, can realize the scanning inventory of user to steel plate stacking.The steel plate inventory method described in the application scans steel plate stacking section using camera, obtains steel plate stacking section complete image according to four pairs of optimal matching feature points of adjacent image frames, improves the distinguishability of steel plate boundary in steel plate stacking section image.Using the coarse extraction of steel plate region to reduce the range where steel plate is located, then taking counting column in the coarse extraction region, finding the upper and lower boundaries of steel plate on each counting column to realize the fine extraction of steel plate region on each counting column.Then marking and counting the steel plate boundary of each counting column, finally obtaining the number of steel plates from the counting results of all counting columns.The steel plate stacking inventory system described in the application provides a convenient steel plate stacking inventory method based on vision by the way of scanning steel plate stacking section using embedded device.

Description

Technical Field

[0001] This invention relates to the fields of embedded systems and vision technology, specifically to a vision-based method and system for counting and inventorying stacked steel plates. Background Technology

[0002] Counting steel plates is a crucial step in the process of steel plates entering and leaving the warehouse. Traditional counting methods include manual counting, thickness-based counting, and weight-based counting. Manual counting is not only time-consuming and labor-intensive, but also lacks accuracy. Thickness-based counting involves measuring the thickness of each individual steel plate and all steel plates, then dividing the total thickness by the thickness of a single plate to determine the quantity. This method is highly dependent on the thickness of the steel plates. Weight-based counting is similar, dividing the total weight of all steel plates by the weight of a single plate. This method is also highly dependent on the weight of the steel plates, and heavier plates increase the workload. A visual method, however, involves capturing side images of the steel plates with a camera and then using image processing to determine the quantity. This method is not only time-saving and labor-saving but also offers high accuracy, better meeting application requirements.

[0003] Directly photographing the side of the steel plates is problematic due to limitations of the camera and the number of plates. When there are many plates, the boundaries between them are blurred in the resulting image, making effective identification and counting impossible. Obtaining an image with clear steel plate boundaries is a prerequisite for accurate steel plate counting in practical applications.

[0004] The accuracy of steel plate stack counting depends on the strategy used to count the steel plates. Currently, there is no fixed procedure for counting steel plates using image processing methods; different methods can be used. One method, based on connected component labeling, involves first converting the acquired image to grayscale, then binarizing and filtering it, and finally counting the number of steel plates by labeling the connected components. However, in practical applications, due to insufficient or excessive ambient light, and oil contamination on the steel plate stack cross-sections, it is difficult to obtain an ideal situation where white areas and black gaps of steel plates are interspersed. Therefore, counting the connected components will not yield an accurate number of steel plates. Thus, choosing the right method to count the number of steel plates in a stack is crucial for accurate and rapid counting. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the purpose of this invention is to provide a vision-based steel plate stacking and counting system, solving the problem of unclear steel plate boundaries in camera images when the number of steel plates is large, and proposing a method for accurately and effectively counting the number of steel plates. This invention uses a handheld embedded device to obtain a complete and clear cross-sectional image of the steel plate stack in real time through camera scanning and image stitching, and then uses steel plate boundary markings based on the cross-sectional image to count the number of steel plates.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A vision-based method for counting and inventorying stacked steel plates, with the following specific steps:

[0008] S1: Image acquisition of the cross-section of the steel plate stack. A handheld embedded device is used to scan the cross-section of the steel plate stack from top to bottom. Images are stitched together using similar portions from adjacent image frames to obtain a complete cross-sectional image of the steel plate stack. The specific steps are as follows:

[0009] S11: Perform sharpness analysis on the image frame. If the sharpness is less than the threshold, remove the frame and continue to capture the next image frame and perform sharpness analysis to complete the removal of blurry image frames during the scanning process.

[0010] S12: For adjacent image frames with a sharpness greater than the threshold, divide them into four equal parts: top left, top right, bottom left, and bottom right. Extract and match feature points for corresponding parts in adjacent image frames. Rank the matched feature points in each of the four parts according to their matching degree, and select the four pairs of points with the best matching degree, which are P's of the first image. 11 (x 11 ,y 11 ), P 12 (x 12 ,y 12 ), P 13 (x 13 ,y 13 ), P 14 (x 14 ,y 14 ) and P in the second picture 21 (x 21 ,y 21 ), P 22 (x 22 ,y 22 ), P 23 (x 23 ,y 23 ), P 24 (x 24 ,y 24 ).

[0011] S13: Further verify the matching accuracy and the existence of similar parts based on the parallelism of the four pairs of points. The judgment method is as follows:

[0012] ||x 2i -x 1i |-|x 21 -x 11 ||<th1,i=2,3,4

[0013] ||y 2i -y 1i |-|y 21 -y 11 ||<th2,i=2,3,4

[0014] Where th1 and th2 are the allowable deviation thresholds between the parallelism of the four pairs of matching points.

[0015] S14: If the parallelism of the four pairs of points satisfies the above relationship, it indicates that there are similar parts in adjacent frames. Then, the positional offset of the second image relative to the first image is calculated based on the four pairs of points, that is, the horizontal and vertical movement between adjacent image frames. The calculation method is as follows:

[0016]

[0017]

[0018] S15: Based on the calculated positional offset, remove image frames with excessive horizontal offset and vertical upward offset, according to the following criteria:

[0019] |Δx|>th3 or Δy<0

[0020] Where th3 is the horizontal deviation threshold between adjacent image frames.

[0021] S16: Copy the first image frame to the middle position of the stitched image S(x,y). Subsequent image frames are stitched and fused based on this. According to the calculated positional offset between adjacent image frames, copy the newly appearing steel plate portion and its 10 pixels above it from the next image frame down relative to the previous image frame to a transformed image W(x,y) of the same size as the stitched image. All pixel values ​​in W(x,y) are 0 except for the copied portion. Then, the stitched image and the transformed image are fused using the following method:

[0022]

[0023] Where α and β are different weights.

[0024] S17: Perform the above operation on subsequent adjacent image frames until the entire cross-sectional image of the steel plate stack is stitched together to obtain a complete and clear cross-sectional image of the steel plate stack.

[0025] S2: Coarse Extraction of Steel Plate Area. After obtaining a complete and clear cross-section of the steel plate stack, coarse extraction of the steel plate area is performed to obtain the approximate area of ​​the steel plates, narrowing the steel plate counting range and reducing the influence of the external environment on the steel plate counting. The specific steps are as follows:

[0026] S21: Edge detection using the Canny operator.

[0027] S22: Morphological processing is performed on the image that has undergone edge detection using the Canny operator. The smallest bounding rectangle of the largest area is found in the morphologically processed image, which gives the approximate area of ​​the steel plate, thus achieving coarse extraction of the steel plate area.

[0028] S3: Refined Extraction of Steel Plate Region. Within the coarsely extracted steel plate region, count columns are taken horizontally. The upper and lower boundaries of the steel plate are found on each count column to achieve refined extraction of the steel plate region on each count column, further narrowing down the area of ​​the steel plate. The specific steps are as follows:

[0029] S31: Perform grayscale processing on the image. Within the coarsely extracted steel plate area, divide the area into n equal blocks horizontally. Take the middle column of pixel values ​​from each block, called the counting column. There are a total of n counting columns.

[0030] S32: Take the derivative of the grayscale value for each count column, as follows:

[0031] value = D(i,j) - D(i+1,j)

[0032] Where value is the derivative of grayscale, D(i,j) is the grayscale value of the i-th row in the j-th column, and D(i+1,j) is the grayscale value of the (i+1)-th row in the j-th column.

[0033] S33: Obtain the boundary of the steel plate based on the extreme value of the gray-level derivative. The gray-level derivative of the steel plate reaches an extreme value at the boundary and is greater than a certain threshold. Based on this condition, possible steel plate boundaries can be found.

[0034] When the derivative value of the grayscale is a maximum, the following conditions are used to determine whether it is considered a possible steel plate boundary:

[0035] value>valueL&&value>valueN&&value>th4

[0036] Where valueL and valueN are the grayscale derivatives adjacent to value, and th4 is the minimum threshold of the grayscale derivative value considered as the boundary of the steel plate when the grayscale derivative value reaches its maximum value.

[0037] The derivative value of the grayscale is considered to be the minimum, and the following conditions are used to determine the possible boundary of the steel plate:

[0038] value<valueL&&value<valueN&&value<th5

[0039] Where valueL and valueN are the grayscale derivatives adjacent to value, and th5 is the maximum threshold of the grayscale derivative value considered as the boundary of the steel plate when the grayscale derivative value reaches its minimum value.

[0040] S34: Calculate the average of the extreme values ​​of the derivative of the grayscale values ​​in each counting column, using the following method:

[0041]

[0042]

[0043] Where avgMax is the average of the maxima, avgMin is the average of the minima, and m and n are the number of maxima and minima, respectively.

[0044] S35: On each counting column, starting from the middle of the rectangle, search upwards and downwards for the upper and lower boundaries of the steel plate. The extreme values ​​are possible boundaries. If the maximum is greater than avgMax or the minimum is less than avgMin, the row number of the extreme value is pre-determined as the possible upper or lower boundary of the steel plate, and the search continues upwards and downwards. If an extreme value that satisfies the above conditions is found, the row number of the possible upper or lower boundary of the steel plate is updated. If several consecutive extreme values ​​do not satisfy the above conditions or there are no extreme values ​​upwards or downwards, the search ends. The previously found possible upper and lower boundaries are taken as the upper and lower boundaries of this column. The area within the upper and lower boundaries is the area of ​​the steel plate in this counting column, thus realizing the fine extraction of the steel plate area on each counting column.

[0045] S4: Steel Plate Quantity Count. Within the upper and lower boundaries of each counting column, mark and count the steel plates to obtain the quantity of steel plates in each column. Then, combine the count results of all counting columns and take the number with the highest count as the quantity of steel plates. The specific steps are as follows:

[0046] S41: Within the upper and lower boundaries of each counting column, all rows containing the maximum are considered as preliminary steel plate boundaries, and the number of pixels between adjacent boundaries is counted. The number of pixels is the thickness of the steel plate in the image. The thickness value that appears most frequently among the steel plate thicknesses calculated from all adjacent boundaries is taken as the average thickness h of the steel plate in this counting column.

[0047] S42: On each counting column, starting from the upper boundary to the lower boundary, use h to adaptively correct for places where the distance d between two adjacent possible boundaries is too large or too small. If d is less than or equal to four-fifths of h, it is not considered a boundary and is not marked. The first boundary remains unchanged, and the search for the nearest boundary continues downwards, with adaptive correction performed. If d is greater than four-fifths of h but less than or equal to 1.5 times h, it is considered a boundary, and its row is marked. If d is greater than 1.5 times h, boundary additions are needed between the boundaries, and the added boundaries are marked. The number of boundary additions is obtained as follows:

[0048] N = [d / h - 0.5]

[0049] S43: Count the steel plates according to the marked boundaries and recalculate the average thickness of the steel plates in this column. Use the newly obtained average thickness to perform adaptive correction again. If the two steel plate counting results are the same, take the result as the number of steel plates in this column. If the two steel plate counting results are different, repeat the above steps until the counting results are the same or a certain number of iterations are reached. Take the final counting result as the number of steel plates in this column.

[0050] S44: The result that appears most frequently among all the counting results in the counting columns is taken as the final number of steel plates, calculated as follows:

[0051] num = argmax{ncount(num k ),k=1,2,…,l}

[0052] Where, ncount(num k ) represents the number of times the k-th counting result appears, and l represents that there are l possible counting results in all counting columns.

[0053] A vision-based steel plate stacking and counting system, based on the aforementioned vision-based steel plate stacking and counting method, includes six modules: a human-computer interaction module, a ranging module, a camera module, a processor module, a light source module, and a power supply module. The human-computer interaction module enables interaction between the user and the embedded device, allowing the user to set the start or end of scanning the steel plate stack cross-section and counting, and also providing the user with relevant device information. The ranging module measures the distance between the device and the steel plate stack cross-section. The camera module scans the steel plate stack cross-section and obtains image frames. The processor module processes the information obtained from the human-computer interaction module, ranging module, and camera module, and performs corresponding operations. The light source module illuminates the steel plate stack cross-section, and the power supply module provides power to the device.

[0054] In this invention, the human-computer interaction module functions as follows: user input of operation information and user reading of device information. User input includes starting and ending the scanning of the steel plate stack cross-section. User reading of device information includes four aspects: viewing the real-time image of the scanned composite, displaying and prompting the distance between the device and the steel plate stack cross-section during scanning, viewing the steel plate boundary markings and the steel plate stack count results, and viewing the remaining battery power of the device.

[0055] In this invention, the ranging module is used to ensure that the distance between the user-controlled device and the steel plate stack cross-section is within a suitable range, resulting in clearer images captured by the camera and facilitating higher-precision image stitching. When using the device, the user can see the recommended distance and the actual distance on the human-machine interface module. When the distance between the device and the steel plate stack cross-section deviates significantly from the recommended distance, the human-machine interface module will prompt the user on how to adjust the device to a suitable distance.

[0056] In this invention, the processor module processes information obtained from the human-computer interaction module, the ranging module, and the camera module. The processor module obtains the distance between the device and the cross-section of the steel plate stack from the ranging module and displays it on the human-computer interaction module. The user obtains the distance information from the human-computer interaction module, adjusts the device to a suitable position, inputs the start scanning information, and the processor module controls the camera to open. The processor module then stitches the image frames obtained from the camera into a single image using an image stitching method and displays the image on the human-computer interaction interface in real time. After receiving the end scanning information, the processor module stops image stitching, counts the number of steel plates in the stitched cross-section diagram of the steel plate stack, and finally displays the result on the human-computer interaction module.

[0057] Compared with the prior art, the present invention has the following advantages:

[0058] This invention realizes a vision-based method and system for counting steel plate stacks. The vision-based method overcomes the time-consuming and labor-intensive drawbacks of traditional manual counting methods, and its accuracy is significantly improved compared to thickness-based and weight-based methods, with a wider range of applications. Furthermore, as a handheld embedded device, it offers convenience. For acquiring cross-sectional images of the steel plate stacks, a camera scanning method is used. The scanned image frames are stitched together to obtain a clear cross-sectional image of the steel plate stack, facilitating steel plate counting. For counting the steel plate stacks, the steel plate area is divided into n equal blocks horizontally. The middle column of pixel values ​​in each block is used as a counting column. Possible steel plate boundaries are found by differentiating the grayscale values ​​in each counting column. After continuous correction, the final boundary is obtained and used for counting. Finally, the count results from multiple counting columns are used to obtain the final number of steel plates, greatly improving accuracy and efficiency. Attached Figure Description

[0059] Figure 1 This is a flowchart of a vision-based steel plate stacking and counting method;

[0060] Figure 2 This is a schematic diagram of the embedded device used in this invention;

[0061] Figure 3 This is a schematic diagram of the matching of four pairs of optimal feature points in adjacent image frames after screening in this invention;

[0062] Figure 4 This is a schematic diagram of the fusion of stitched and transformed images in this invention;

[0063] Figure 5 This is a schematic diagram of the counting sequence of the coarse extraction steel plate area in this invention;

[0064] Figure 6 This is a system framework diagram of the embedded device in this invention;

[0065] Figure 7 This is a flowchart illustrating the process of using embedded devices for steel plate stacking and inventory in this invention. Detailed Implementation

[0066] The specific implementation of the present invention will be further described in detail below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, a vision-based method for counting stacked steel plates includes the following steps:

[0068] S1: Obtaining the cross-sectional image of the steel plate stack. For example... Figure 2As shown, a handheld embedded device is used to scan the cross-section of the steel plate stack from top to bottom. Images are stitched together using similar parts in adjacent image frames to obtain a complete cross-sectional image of the steel plate stack. The specific steps are as follows:

[0069] S11: Perform sharpness analysis on the image frame. If the sharpness is less than the threshold, remove the frame and continue to capture the next image frame and perform sharpness analysis to complete the removal of blurry image frames during the scanning process.

[0070] S12: For adjacent image frames with a sharpness greater than the threshold, divide them into four equal parts: top left, top right, bottom left, and bottom right. Extract and match feature points for corresponding parts in adjacent image frames. Rank the matched feature points in each of the four parts according to their matching degree, and select the four pairs of points with the best matching degree, which are P's of the first image. 11 (x 11 ,y 11 ), P 12 (x 12 ,y 12 ), P 13 (x 13 ,y 13 ), P 14 (x 14 ,y 14 ) and P in the second picture 21 (x 21 ,y 21 ), P 22 (x 22 ,y 22 ), P 23 (x 23 ,y 23 ), P 24 (x 24 ,y 24 ),like Figure 3 As shown.

[0071] S13: Further verify the matching accuracy and the existence of similar parts based on the parallelism of the four pairs of points. The judgment method is as follows:

[0072] ||x 2i -x 1i |-|x 21 -x 11 ||<th1,i=2,3,4

[0073] ||y 2i -y 1i |-|y 21 -y 11 ||<th2,i=2,3,4

[0074] Where th1 and th2 are the allowable deviation thresholds between the parallelism of the four pairs of matching points.

[0075] S14: If the parallelism of the four pairs of points satisfies the above relationship, it indicates that there are similar parts in adjacent frames. Then, the positional offset of the second image relative to the first image is calculated based on the four pairs of points, that is, the horizontal and vertical movement between adjacent image frames. The calculation method is as follows:

[0076]

[0077]

[0078] S15: Based on the calculated positional offset, remove image frames with excessive horizontal offset and vertical upward offset, according to the following criteria:

[0079] |Δx|>th3 or Δy<0

[0080] Where th3 is the horizontal deviation threshold between adjacent image frames.

[0081] S16: As Figure 4 As shown, the first image frame is copied to the middle position of the stitched image S(x,y), and subsequent image frames are stitched and fused based on this. Based on the calculated positional offset between adjacent image frames, the newly appearing steel plate portion and its 10 pixels above it in the next image frame are copied to a transformed image W(x,y) of the same size as the stitched image. All pixel values ​​in W(x,y) are 0 except for the copied portion. Then, the stitched image and the transformed image are fused using the following method:

[0082]

[0083] Where α and β are different weights.

[0084] S17: Perform the above operation on subsequent adjacent image frames until the entire cross-sectional image of the steel plate stack is stitched together to obtain a complete and clear cross-sectional image of the steel plate stack.

[0085] S2: Coarse Extraction of Steel Plate Area. After obtaining a complete and clear cross-section of the steel plate stack, coarse extraction of the steel plate area is performed to obtain the approximate area of ​​the steel plates, narrowing the steel plate counting range and reducing the influence of the external environment on the steel plate counting. The specific steps are as follows:

[0086] S21: Edge detection using the Canny operator.

[0087] S22: Morphological processing is performed on the image that has undergone edge detection using the Canny operator. The smallest bounding rectangle of the largest area is found in the morphologically processed image, which gives the approximate area of ​​the steel plate, thus achieving coarse extraction of the steel plate area.

[0088] S3: Refined Extraction of Steel Plate Region. Within the coarsely extracted steel plate region, count columns are taken horizontally. The upper and lower boundaries of the steel plate are found on each count column to achieve refined extraction of the steel plate region on each count column, further narrowing down the area of ​​the steel plate. The specific steps are as follows:

[0089] S31: Perform grayscale processing on the image. Within the coarsely extracted steel plate area, such as... Figure 5 As shown, the region is divided into n equal blocks horizontally. The middle column of pixel values ​​in each block is called the counting column. There are a total of n counting columns.

[0090] S32: Take the derivative of the grayscale value for each count column, as follows:

[0091] value = D(i,j) - D(i+1,j)

[0092] Where value is the derivative of grayscale, D(i,j) is the grayscale value of the i-th row in the j-th column, and D(i+1,j) is the grayscale value of the (i+1)-th row in the j-th column.

[0093] S33: Obtain the boundary of the steel plate based on the extreme value of the gray-level derivative. The gray-level derivative of the steel plate reaches an extreme value at the boundary and is greater than a certain threshold. Based on this condition, possible steel plate boundaries can be found.

[0094] When the derivative value of the grayscale is a maximum, the following conditions are used to determine whether it is considered a possible steel plate boundary:

[0095] value>valueL&&value>valueN&&value>th4

[0096] Where valueL and valueN are the grayscale derivatives adjacent to value, and th4 is the minimum threshold of the grayscale derivative value considered as the boundary of the steel plate when the grayscale derivative value reaches its maximum value.

[0097] The derivative value of the grayscale is considered to be the minimum, and the following conditions are used to determine the possible boundary of the steel plate:

[0098] value<valueL&&value<valueN&&value<th5

[0099] Where valueL and valueN are the grayscale derivatives adjacent to value, and th5 is the maximum threshold of the grayscale derivative value considered as the boundary of the steel plate when the grayscale derivative value reaches its minimum value.

[0100] S34: Calculate the average of the extreme values ​​of the derivative of the grayscale values ​​in each counting column, using the following method:

[0101]

[0102]

[0103] Where avgMax is the average of the maxima, avgMin is the average of the minima, and m and n are the number of maxima and minima, respectively.

[0104] S35: On each counting column, starting from the middle of the rectangle, search upwards and downwards for the upper and lower boundaries of the steel plate. The extreme values ​​are possible boundaries. If the maximum is greater than the maximum or the minimum is less than the minimum, then the row number of the maximum or minimum is pre-determined as a possible upper or lower boundary of the steel plate, and the search continues upwards and downwards. If an extreme value that satisfies the above conditions is found, then the row number of the possible upper or lower boundary of the steel plate is updated. If several consecutive extreme values ​​do not satisfy the above conditions or there are no extreme values ​​upwards or downwards, then the search ends. The previously found possible upper and lower boundaries are taken as the upper and lower boundaries of this column. The area within the upper and lower boundaries is the area of ​​the steel plate in this counting column, thus realizing the fine extraction of the steel plate area on each counting column.

[0105] S4: Steel Plate Quantity Count. Within the upper and lower boundaries of each counting column, mark and count the steel plates to obtain the quantity of steel plates in each column. Then, combine the count results of all counting columns and take the number with the highest count as the quantity of steel plates. The specific steps are as follows:

[0106] S41: Within the upper and lower boundaries of each counting column, all rows containing the maximum are considered as preliminary steel plate boundaries, and the number of pixels between adjacent boundaries is counted. The number of pixels is the thickness of the steel plate in the image. The thickness value that appears most frequently among the steel plate thicknesses calculated from all adjacent boundaries is taken as the average thickness h of the steel plate in this counting column.

[0107] S42: On each counting column, starting from the upper boundary to the lower boundary, use h to adaptively correct for places where the distance d between two adjacent possible boundaries is too large or too small. If d is less than or equal to four-fifths of h, it is not considered a boundary and is not marked. The first boundary remains unchanged, and the search for the nearest boundary continues downwards, with adaptive correction performed. If d is greater than four-fifths of h but less than or equal to 1.5 times h, it is considered a boundary, and its row is marked. If d is greater than 1.5 times h, boundary additions are needed between the boundaries, and the added boundaries are marked. The number of boundary additions is obtained as follows:

[0108] N = [d / h - 0.5]

[0109] S43: Count the steel plates according to the marked boundaries and recalculate the average thickness of the steel plates in this column. Use the newly obtained average thickness to perform adaptive correction again. If the two steel plate counting results are the same, take the result as the number of steel plates in this column. If the two steel plate counting results are different, repeat the above steps until the counting results are the same or a certain number of iterations are reached. Take the final counting result as the number of steel plates in this column.

[0110] S44: The result that appears most frequently among all the counting results in the counting columns is taken as the final number of steel plates, calculated as follows:

[0111] num = argmax{ncount(num k ),k=1,2,…,l}

[0112] Where, ncount(num k ) represents the number of times the k-th counting result appears, and l represents that there are l possible counting results in all counting columns.

[0113] A vision-based steel plate stacking and counting system, based on the aforementioned vision-based steel plate stacking and counting method, such as... Figure 6 As shown, the device comprises six modules: a human-computer interaction module, a ranging module, a camera module, a processor module, a light source module, and a power supply module. The human-computer interaction module enables interaction between the user and the embedded device. It is used to set the start and end times for scanning the steel plate stack cross-section and to count data. It also allows the user to obtain relevant information about the device. The ranging module measures the distance between the device and the steel plate stack cross-section. The camera module scans the steel plate stack cross-section and obtains image frames. The processor module processes the information obtained from the human-computer interaction module, the ranging module, and the camera module, and performs corresponding operations. The light source module illuminates the steel plate stack cross-section, and the power supply module provides power to the device.

[0114] In this invention, the human-computer interaction module functions as follows: user input of operation information and user reading of device information. User input includes starting and ending the scanning of the steel plate stack cross-section. User reading of device information includes four aspects: viewing the real-time image of the scanned composite, displaying and prompting the distance between the device and the steel plate stack cross-section during scanning, viewing the steel plate boundary markings and the steel plate stack count results, and viewing the remaining battery power of the device.

[0115] In this invention, the ranging module is used to ensure that the distance between the user-controlled device and the steel plate stack cross-section is within a suitable range, resulting in clearer images captured by the camera and facilitating higher-precision image stitching. When using the device, the user can see the recommended distance and the actual distance on the human-machine interface module. When the distance between the device and the steel plate stack cross-section deviates significantly from the recommended distance, the human-machine interface module will prompt the user on how to adjust the device to a suitable distance.

[0116] In this invention, the processor module processes information obtained from the human-computer interaction module, the ranging module, and the camera module. The processor module obtains the distance between the device and the cross-section of the steel plate stack from the ranging module and displays it on the human-computer interaction module. The user obtains the distance information from the human-computer interaction module, adjusts the device to a suitable position, inputs the start scanning information, and the processor module controls the camera to open. The processor module then stitches the image frames obtained from the camera into a single image using an image stitching method and displays the image on the human-computer interaction interface in real time. After receiving the end scanning information, the processor module stops image stitching, counts the number of steel plates in the stitched cross-section diagram of the steel plate stack, and finally displays the result on the human-computer interaction module.

[0117] In this invention, the process of stacking and counting embedded devices using steel plates is as follows: Figure 7 As shown, after turning on the device, the user first adjusts the device to a suitable position according to the information on the human-machine interface module, and then confirms to start scanning the cross-section of the steel plate stack. The user scans the cross-section of the steel plate stack from top to bottom, during which the human-machine interface module displays the stitched image in real time. After the scan is completed, the user confirms to end the scan. Finally, the device outputs the inventory results.

[0118] The detailed descriptions listed in this invention are merely specific descriptions of feasible embodiments of the invention and are not intended to limit the scope of protection of the invention. All equivalent embodiments or modifications made without departing from the spirit of the invention should be included within the scope of protection of the invention.

Claims

1. A vision-based method for counting and inventorying stacked steel plates, characterized in that, Includes the following steps: S1: Acquisition of cross-sectional image of steel plate stack; The cross-sectional image of the steel plate stack is scanned from top to bottom using a handheld embedded device, and the images are stitched together using similar parts in adjacent image frames to obtain a complete cross-sectional image of the steel plate stack. Step S1 includes the following steps: S11: Perform sharpness analysis on the image frame. If the sharpness is less than the threshold, remove the frame and continue to capture the next image frame and perform sharpness analysis to complete the removal of blurry image frames during the scanning process. S12: For adjacent image frames with a sharpness greater than the threshold, divide them into four equal parts: top left, top right, bottom left, and bottom right. Extract and match feature points for corresponding parts in adjacent image frames. Sort the matching feature points in the four parts by matching degree and select the four pairs of points with the best matching degree. S13: Further verify whether the matching is correct and whether there are similar parts based on the parallelism of the four pairs of points; S14: If the parallelism of the four pairs of points satisfies the above relationship, it means that there are similar parts in adjacent frames. Then, the positional offset of the second image relative to the first image is obtained based on the four pairs of points, that is, the horizontal and vertical movement between adjacent image frames. S15: Based on the calculated positional offset, remove image frames that have excessive horizontal offset or vertical upward offset; S16: Copy the first image frame to the position in the stitched image, and then stitch and merge subsequent image frames based on this position. S17: Perform the above operation on subsequent adjacent image frames until the entire cross-sectional image of the steel plate stack is stitched together to obtain a complete and clear cross-sectional image of the steel plate stack. S2: Coarse extraction of steel plate area; After obtaining a complete and clear cross section of the steel plate stack, coarse extraction of the steel plate area is performed to obtain the approximate area of ​​the steel plate, thereby narrowing the steel plate counting range and reducing the influence of the external environment on the steel plate counting. S3: Fine extraction of steel plate area; Within the coarsely extracted steel plate area, take a counting column in the horizontal direction, and find the upper and lower boundaries of the steel plate on each counting column to achieve fine extraction of the steel plate area on each counting column, further narrowing down the range of the steel plate; S4: Steel plate quantity statistics; Within the upper and lower boundary range of each counting column, mark and count the steel plate boundaries to obtain the quantity of steel plates in each counting column. Then, combine the counting results of all counting columns and take the number with the most counts as the quantity of steel plates, thereby counting the quantity of steel plates.

2. The vision-based steel plate stacking and counting method according to claim 1, characterized in that, Step S1 includes the following steps: S11: Perform sharpness analysis on the image frame. If the sharpness is less than the threshold, remove the frame and continue to capture the next image frame and perform sharpness analysis to complete the removal of blurry image frames during the scanning process. S12: For adjacent image frames with a sharpness greater than the threshold, divide them into four equal parts: top left, top right, bottom left, and bottom right. Extract and match feature points for corresponding parts in adjacent image frames. Rank the matched feature points in each of the four parts according to their matching degree, and select the four pairs of points with the best matching degree, which are then used as the first image's references. The second picture ; S13: Further verify the matching accuracy and the existence of similar parts based on the parallelism of the four pairs of points. The judgment method is as follows: ； in, , This is the allowable deviation threshold between the parallelism of four pairs of matching points; S14: If the parallelism of the four pairs of points satisfies the above relationship, it indicates that there are similar parts in adjacent frames. Then, the positional offset of the second image relative to the first image is calculated based on the four pairs of points, that is, the horizontal and vertical movement between adjacent image frames. The calculation method is as follows: ； S15: Based on the calculated positional offset, remove image frames with excessive horizontal offset and vertical upward offset, according to the following criteria: |Δx|>th3 or Δy<0; in, The horizontal deviation threshold between adjacent image frames; S16: Copy the first image frame to the stitched image. The upper and middle positions are used as the basis for subsequent image frames to be stitched and fused. Based on the positional offset value calculated between adjacent image frames, the newly appearing steel plate portion and the portion 10 pixels above it in the next frame relative to the previous frame are copied to a transformed image of the same size as the stitched image. middle, Except for the copied portion, all other pixel values ​​are set to 0. Then, the stitched image and the transformed image are merged. The merging method is as follows: ； in, α , β For different weights; S17: Perform the above operation on subsequent adjacent image frames until the entire cross-sectional image of the steel plate stack is stitched together to obtain a complete and clear cross-sectional image of the steel plate stack.

3. The vision-based steel plate stacking and counting method according to claim 1, characterized in that, Step S2 includes the following steps: S21: Edge detection using the Canny operator; S22: Morphological processing is performed on the image that has undergone edge detection using the Canny operator. The smallest bounding rectangle of the largest area is found in the morphologically processed image, which gives the approximate area of ​​the steel plate, thus achieving coarse extraction of the steel plate area.

4. The vision-based steel plate stacking and counting method according to claim 1, characterized in that, Step S3 includes the following steps: S31: Perform grayscale processing on the image. Within the coarsely extracted steel plate area, divide the area into n equal blocks in the horizontal direction. Take the middle column of pixel values ​​of each block, which is called the counting column. There are a total of n counting columns. S32: Take the derivative of the grayscale value for each count column, as follows: ； in, Find the derivative of the grayscale value. For the first j List of the first i The grayscale value of the row, For the first j List of the first i + The grayscale values ​​of line 1; S33: Obtain the boundary of the steel plate based on the extreme value of the gray-level derivative; the gray-level derivative of the steel plate reaches an extreme value at the boundary of the steel plate and is greater than a certain threshold. Based on this condition, find possible steel plate boundaries. Gray-scale derivative Maximum value When a steel plate boundary is considered a possible boundary, the following conditions are met: ； in, To and The derivative of adjacent gray levels, The minimum threshold for the grayscale derivative value is considered as the boundary of the steel plate when the grayscale derivative value reaches its maximum value. Gray-scale derivative Minimum value When a steel plate boundary is considered a possible boundary, the following conditions are met: ； in, To and The derivative of adjacent gray levels, The maximum threshold value of the grayscale derivative value is considered as the boundary of the steel plate when the grayscale derivative value reaches its minimum value; S34: Calculate the average of the extreme values ​​of the derivative of the grayscale values ​​in each counting column, using the following method: ； in, The average of the maximum values, The average of the minimum values, m , n These represent the number of maxima and minima, respectively. S35: In each counting column, starting from the middle of the rectangle, find the upper and lower boundaries of the steel plate, respectively, upwards and downwards. The extreme values ​​are the possible boundaries. Greater than or Less than If the row number is pre-determined as a possible upper or lower boundary of the steel plate, the search continues upward and downward. If an extreme value that satisfies the above conditions is found, the row number of the possible upper or lower boundary of the steel plate is updated. If several consecutive extreme values ​​do not satisfy the above conditions or there are no extreme values ​​upward or downward, the search ends. The previously found possible upper and lower boundaries are taken as the upper and lower boundaries of this column. The area within the upper and lower boundaries is the area of ​​the steel plate in this counting column, thus realizing the fine extraction of the steel plate area on each counting column.

5. The vision-based steel plate stacking and counting method according to claim 1, characterized in that, Step S4 includes the following steps: S41: Within the upper and lower boundaries of each counting column, all The row number is considered as the initial steel plate boundary, and the number of pixels between adjacent boundaries is counted. The number of pixels is the thickness of the steel plate in the image. The thickness value that appears most frequently among the thicknesses of each layer of steel plate calculated from all adjacent boundaries is taken as the average thickness of the steel plate in this count column. h ; S42: On each counting column, starting from the top boundary to the bottom boundary, using... h To adaptively correct the distance between two adjacent possible boundaries in turn. d A place that is too large or too small; if d Less than or equal to h If four-fifths of it is not considered a boundary, it is not marked, the first boundary remains unchanged, and the search for the nearest boundary continues downward with adaptive correction. like d Greater than h Four-fifths but less than or equal to h If it is 1.5 times the value, it is considered a boundary, and its row is marked; if d Greater than h If the value is 1.5 times the original value, then boundary additions are needed between the boundaries, and the added boundaries need to be marked. The number of boundary additions is obtained from the following: ； S43: Count the steel plates according to the marked boundaries and recalculate the average thickness of the steel plates in this column. Use the newly obtained average thickness to perform adaptive correction again. If the two steel plate counting results are the same, take the result as the number of steel plates in this column. If the two steel plate counting results are different, repeat the above steps until the counting results are the same or a certain number of iterations are reached. Take the final counting result as the number of steel plates in this column. S44: The result that appears most frequently among all the counting results in the counting columns is taken as the final number of steel plates, calculated as follows: ； in, Indicates the first k The number of times the count result appears, l This indicates the total number of counts for all columns. l kind.

6. A vision-based steel plate stacking and counting system, based on the vision-based steel plate stacking and counting method of claim 1, characterized in that, It includes a human-computer interaction module, a ranging module, a camera module, a processor module, a light source module, and a power supply module. The human-computer interaction module enables interaction between the user and the embedded device, serving two purposes: setting the start or end of scanning the steel plate stack cross-section and counting, and providing the user with relevant device information. The ranging module measures the distance between the device and the steel plate stack cross-section. The camera module scans the steel plate stack cross-section and obtains image frames. The processor module processes the information obtained from the human-computer interaction module, ranging module, and camera module, and performs corresponding operations. The light source module illuminates the steel plate stack cross-section, and the power supply module provides power to the device.

7. The vision-based steel plate stacking and inventory system according to claim 6, characterized in that, The human-computer interaction module has the following functions: inputting user operation information and reading device information. The user operation information input is the user inputting to start scanning the steel plate stack cross section and the user inputting to end scanning the steel plate stack cross section. The user reading of device information is divided into four aspects: viewing the real-time image of the scan splicing, displaying and prompting the distance between the device and the steel plate stack cross section during scanning, viewing the steel plate boundary marking map and the steel plate stack count results, and viewing the remaining power of the device.

8. The vision-based steel plate stacking and inventory system according to claim 6, characterized in that, The ranging module is used to keep the distance between the user-controlled device and the steel plate stack section within a suitable range, so that the images captured by the camera are clearer and facilitate higher-precision image stitching. When using the device, the user can see the recommended distance and the actual distance in the human-computer interaction module. When the distance between the device and the steel plate stack section deviates significantly from the recommended distance, the human-computer interaction module will prompt the user on how to adjust the device to a suitable distance.

9. The vision-based steel plate stacking and inventory system according to claim 6, characterized in that, The processor module processes information obtained from the human-computer interaction module, the ranging module, and the camera module. The processor module obtains the distance between the device and the steel plate stack cross-section from the ranging module and displays it on the human-computer interaction module. The user obtains the distance information from the human-computer interaction module, adjusts the device to a suitable position, inputs the start scanning information, and the processor module controls the camera to open. The processor module then stitches the image frames obtained from the camera into a single image using an image stitching method and displays the image on the human-computer interaction interface. After receiving the end scanning information, the processor module stops image stitching, counts the stitched steel plate stack cross-section, and finally displays the result on the human-computer interaction module.

Images