Steel long material image stack position target detection and automatic sorting method based on RetinaNet algorithm

Abstract

The invention relates to a steel long material image stack position target detection and automatic sorting method based on a RetinaNet algorithm, and belongs to the technical field of machine vision target detection. The method comprises the five steps of image acquisition and marking, construction of a target detection model, calculation of midpoint coordinates of each steel stack position, calculation of the row number position and the column number position of each steel stack position, marking and display. By means of the method, the target number and position of the stacking positions of the warehouse steel stacking image can be automatically calculated, and automatic stocktaking of steel stacking in various warehouse complex environments can be achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine vision target detection, and particularly relates to a steel long material image stack position target detection and automatic sorting method based on a RetinaNet algorithm. BACKGROUND

[0002] Traditional warehouse management has many pain points in inventory taking, such as being mainly artificial, goods being misplaced, missing goods, high labor costs, and low operational efficiency.

[0003] According to requirements and market research, in the field of steel long materials, the warehouse area is mainly manually inventoried, especially when checking the materials placed at high positions, which is done by climbing ladders or personnel climbing the stack position, which is unsafe at high altitudes and has low efficiency and a high error rate. For most small and medium-sized long material production enterprises, the construction of an unmanned warehouse area based on a crown block, ground positioning, and integrated warehouse area requires high investment and enterprise information infrastructure.

[0004] The mainstream technology at the present stage is manual inventory taking and recording in a form, but it is inefficient and prone to errors; bar code and RFID technology, which labels each steel or batch with a label and scans it with a handheld device to automatically input the system, but it is inefficient due to manual scanning; laser scanning or 3D modeling, which obtains three-dimensional data of the steel pile through a laser scanner to calculate the volume and quantity, but the laser scanning and 3D modeling technology is difficult and costly; geomagnetic positioning or RFID positioning, which combines positioning technology to determine the position and movement trajectory of the steel, facilitating inventory taking and tracking, but most warehouse environments are complex and the steel cannot be positioned; unmanned aerial vehicle inspection, but this technology is only suitable for outdoor environment inventory taking.

[0005] Therefore, how to overcome the deficiencies of the prior art is a problem that needs to be solved in the field of machine vision target detection technology at present. SUMMARY

[0006] The purpose of the present application is to overcome the deficiencies of the prior art and provide a steel long material image stack position target detection and automatic sorting method based on a RetinaNet algorithm, which can accurately identify the number of steel long material stack positions and label the steel material stack position rows and columns to realize intelligent inventory taking of the steel warehouse.

[0007] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0008] A steel long material image stack position target detection and automatic sorting method based on a RetinaNet algorithm, comprising the following steps:

[0009] Step (1), collecting images of steel piles in the same type of steel warehouse as the steel pile to be identified, and labeling the stack position coordinates of each steel;

[0010] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0011] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates (x, y) of each steel stack.

[0012] Step (4): Based on the coordinates of the midpoint of the steel stack obtained in step (3), calculate the row and column positions of each steel stack.

[0013] Step (5): Based on the calculation results in step (4), the steel stacks in the image are labeled with rows and columns, and then the row and column positions of the steel stacks are displayed visually.

[0014] Furthermore, in step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

[0015] Furthermore, in step (2), the ratio of the training set to the test set is 8:2 during training and testing.

[0016] Furthermore, in step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows:

[0017] The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive.

[0018] The top left corner coordinate (x) of the first steel stack 11 y 11 The coordinates of the lower right corner are (x 12 y 12 );

[0019] The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 );

[0020] The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 );

[0021]

[0022] ...

[0023]

[0024] Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack;

[0025] Thus, the coordinates of the midpoints of all steel stacks are obtained as set X:

[0026] X = (X1, X2, X3, ..., X...) n ).

[0027] Furthermore, the specific method for step (4) is as follows:

[0028] (1) Set the threshold β between rows:

[0029]

[0030] Where n is the total number of steel stack locations; the coordinate of the upper left corner of the (n-1)th steel stack location (x... (n-1)1 y (n-1)1 The coordinates of the lower right corner are (x (n-1)2 y (n-1)2 );

[0031] (2) Calculate the positional relationship between the coordinates of the midpoints of the stack to determine whether to divide into rows:

[0032] Calculate the difference l between the y-coordinates of the midpoints of two adjacent steel stacks:

[0033] ......

[0035]

[0036] Where l1 is the difference between the y-value of the midpoint of the second steel stack and the y-value of the midpoint of the first steel stack, and l2 is the difference between the y-value of the midpoint of the third steel stack and the y-value of the midpoint of the second steel stack. n-1 This is the difference between the y-value of the midpoint of the nth steel stack and the y-value of the midpoint of the (n-1)th steel stack.

[0037] Then, a judgment is made: the difference in the y-coordinates of the midpoints of two adjacent steel stacks is greater than... Then, a line break is made between these two steel stacks, that is, the steel stack to which the minuend of the difference is calculated is moved to the next line, while the steel stack to which the subtrahend of the difference is calculated is kept in the current line;

[0038] Specifically:

[0039] If it exists Then, determine that the i-th steel stack belongs to the current row, and the (i+1)-th steel stack belongs to the next row; continue this process to obtain m rows.

[0040] (3) Sort the x-values ​​of the midpoint coordinates of the steel stack in each row from smallest to largest, and then assign them a serial number; the serial number starts from 1 and is a positive integer; the serial number is the column number corresponding to the steel stack.

[0041] In step (1) of the present invention, it is preferred to use a target detection and annotation tool to annotate the coordinates of each steel stack, but it is not limited to this and can also be annotated manually.

[0042] In step (1) of this invention, the same type of steel stacks to be identified means that the steel types are the same (e.g., they are all steel pipe stacks) and the stacking methods are the same.

[0043] The RetinaNet model used in this invention is an existing model.

[0044] This invention introduces advanced visual AI technology to achieve intelligent identification, automatic inventory, and data association and analysis of goods.

[0045] This invention's method can be implemented in a WMS system, significantly reducing enterprise investment and making it more attractive to small and medium-sized enterprises. After identifying the layers, rows, and columns of stacked materials using this method, it can be associated with basic product information (such as model, specifications, batch number, etc.) and product process information (such as quality, production process, etc.) for subsequent analysis and utilization.

[0046] Compared with the prior art, the beneficial effects of this invention are as follows:

[0047] Currently, inventory management of long steel products in warehouses is mainly done manually, which is costly and inefficient. This invention can be quickly implemented in various long steel product warehouses, and the target detection model can be continuously iterated as it is used in the field, improving detection accuracy and model generalization ability to meet the needs of intelligent inventory management in various complex steel warehouse environments.

[0048] This invention can realize the inventory of long steel products based on computer vision AI technology, and meet the intelligent inventory of steel stacking in complex environments.

[0049] The method of this invention can improve the efficiency of inventorying long steel products in warehouses, reduce labor and operating costs, and improve the operational efficiency and intelligence level of enterprises. At the same time, with the continuous development of AI technology, the accumulated visual data and models also provide a foundation for the future development of intelligent warehouses.

[0050] Compared with technologies such as manual inventory, barcode RFID, laser scanning, 3D modeling, geomagnetic positioning, and RFID positioning, this invention combines efficiency and accuracy while reducing inventory costs. Attached Figure Description

[0051] Figure 1 The flowchart obtained for the target detection model;

[0052] Figure 2 A 3D model diagram of the algorithm principle structure of the existing RetinaNet;

[0053] Figure 3 A 2D schematic diagram of the decomposition of the existing RetinaNet algorithm 3D model;

[0054] Figure 4 This is a diagram of the steel stacking before the annotation of an application example of this invention;

[0055] Figure 5 This is a diagram of steel stacking after annotation, representing an application example of the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the embodiments.

[0057] Those skilled in the art will understand that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention. Where specific techniques or conditions are not specified in the embodiments, they are performed in accordance with the techniques or conditions described in the literature in the field or according to the product instructions. Materials or equipment whose manufacturers are not specified are all conventional products that can be obtained by purchase.

[0058] Example 1

[0059] A method for target detection and automatic sorting of stacked steel long product images based on the RetinaNet algorithm includes the following steps:

[0060] Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location;

[0061] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0062] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates (x, y) of each steel stack.

[0063] Step (4): Based on the coordinates of the midpoint of the steel stack obtained in step (3), calculate the row and column positions of each steel stack.

[0064] Step (5): Based on the calculation results in step (4), the steel stacks in the image are labeled with rows and columns, and then the row and column positions of the steel stacks are displayed visually.

[0065] Example 2

[0066] A method for target detection and automatic sorting of stacked steel long product images based on the RetinaNet algorithm includes the following steps:

[0067] Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location;

[0068] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0069] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates (x, y) of each steel stack.

[0070] Step (4): Based on the coordinates of the midpoint of the steel stack obtained in step (3), calculate the row and column positions of each steel stack.

[0071] Step (5): Based on the calculation results in step (4), the steel stacks in the image are labeled with rows and columns, and then the row and column positions of the steel stacks are displayed visually.

[0072] In step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

[0073] In step (2), the ratio of training set to test set samples is 8:2 during training and testing.

[0074] In step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows:

[0075] The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive.

[0076] The top left corner coordinate (x) of the first steel stack11 y 11 The coordinates of the lower right corner are (x 12 y 12 );

[0077] The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 );

[0078] The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 );

[0079]

[0080] ...

[0081]

[0082] Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack;

[0083] Thus, the coordinates of the midpoints of all steel stacks are obtained as set X:

[0084] X = (X1, X2, X3, ..., X...) n ).

[0085] The specific method for step (4) is as follows:

[0086] (1) Set the threshold β between rows:

[0087]

[0088] Where n is the total number of steel stack locations; the coordinate of the upper left corner of the (n-1)th steel stack location (x... (n-1)1 y (n-1)1 The coordinates of the lower right corner are (x (n-1)2 y (n-1)2 );

[0089] (2) Calculate the positional relationship between the coordinates of the midpoints of the stack to determine whether to divide into rows:

[0090] Calculate the difference l between the y-coordinates of the midpoints of two adjacent steel stacks:

[0091] ......

[0093]

[0094] Where l1 is the difference between the y-value of the midpoint of the second steel stack and the y-value of the midpoint of the first steel stack, and l2 is the difference between the y-value of the midpoint of the third steel stack and the y-value of the midpoint of the second steel stack. n-1 This is the difference between the y-value of the midpoint of the nth steel stack and the y-value of the midpoint of the (n-1)th steel stack.

[0095] Then, a judgment is made: the difference in the y-coordinates of the midpoints of two adjacent steel stacks is greater than... Then, a line break is made between these two steel stacks, that is, the steel stack to which the minuend of the difference is calculated is moved to the next line, while the steel stack to which the subtrahend of the difference is calculated is kept in the current line;

[0096] (3) Sort the x-values ​​of the midpoint coordinates of the steel stack in each row from smallest to largest, and then assign them a serial number; the serial number starts from 1 and is a positive integer; the serial number is the column number corresponding to the steel stack.

[0097] Example 3

[0098] A method for target detection and automatic sorting of stacked steel long product images based on the RetinaNet algorithm includes the following steps:

[0099] Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location;

[0100] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0101] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates (x, y) of each steel stack.

[0102] Step (4): Based on the coordinates of the midpoint of the steel stack obtained in step (3), calculate the row and column positions of each steel stack.

[0103] Step (5): Based on the calculation results in step (4), the steel stacks in the image are labeled with rows and columns, and then the row and column positions of the steel stacks are displayed visually.

[0104] In step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

[0105] In step (2), the ratio of training set to test set samples is 8:2 during training and testing.

[0106] In step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows:

[0107] The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive.

[0108] The top left corner coordinate (x) of the first steel stack 11 y 11 The coordinates of the lower right corner are (x 12 y 12 );

[0109] The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 );

[0110] The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 );

[0111]

[0112] ...

[0113]

[0114] Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack;

[0115] Thus, the coordinates of the midpoints of all steel stacks are obtained as set X:

[0116] X = (X1, X2, X3, ..., X...) n ).

[0117] The specific method for step (4) is as follows:

[0118] (1) Set the threshold β between rows:

[0119]

[0120] Where n is the total number of steel stack locations; the coordinate of the upper left corner of the (n-1)th steel stack location (x...(n-1)1 y (n-1)1 The coordinates of the lower right corner are (x (n-1)2 y (n-1)2 );

[0121] (2) Calculate the positional relationship between the coordinates of the midpoints of the stack to determine whether to divide into rows:

[0122] Calculate the difference l between the y-coordinates of the midpoints of two adjacent steel stacks:

[0123] ......

[0125]

[0126] Where l1 is the difference between the y-value of the midpoint of the second steel stack and the y-value of the midpoint of the first steel stack, and l2 is the difference between the y-value of the midpoint of the third steel stack and the y-value of the midpoint of the second steel stack. n-1 This is the difference between the y-value of the midpoint of the nth steel stack and the y-value of the midpoint of the (n-1)th steel stack.

[0127] Then, a judgment is made: the difference in the y-coordinates of the midpoints of two adjacent steel stacks is greater than... Then, a line break is made between these two steel stacks, that is, the steel stack to which the minuend of the difference is calculated is moved to the next line, while the steel stack to which the subtrahend of the difference is calculated is kept in the current line;

[0128] If it exists Then, determine that the i-th steel stack belongs to the current row, and the (i+1)-th steel stack belongs to the next row; continue this process to obtain m rows.

[0129] (3) Sort the x-values ​​of the midpoint coordinates of the steel stack in each row from smallest to largest, and then assign them a serial number; the serial number starts from 1 and is a positive integer; the serial number is the column number corresponding to the steel stack.

[0130] For example, if there are 10 steel stacks in the first row, the x-values ​​of the points are sorted from smallest to largest, and then assigned serial numbers 1 to 10 according to the sorting order. The serial number is the column number corresponding to the steel stack. For example, the serial number of the first stack is 1, which means its column number is 1.

[0131] Application Examples

[0132] 1) Collect images of steel inventory in the warehouse using image acquisition equipment and upload them to the system, using a sufficient number of images as the training dataset.

[0133] 2) Traverse all images and use object detection annotation tools to annotate the steel stacks in the sample dataset to obtain the corresponding label files. The label files and the original images together form the image dataset. Randomly select 80% of the sample set as the training set and 20% as the test set.

[0134] 3) Train the RetinaNet model using the training dataset, and test the trained RetinaNet model using the test set to obtain the object detection model (the process is as follows). Figure 1 As shown, the principle is as follows Figure 2 and Figure 3 (As shown).

[0135] 4) Obtain the coordinate values ​​of the test data through the target detection model, calculate the coordinates of the midpoint of the steel stack, and calculate the row and column positions of each steel stack based on the obtained midpoint coordinates of the steel stack.

[0136] 5) Based on the calculated row and column positions of each steel stack, label the steel stacks in the image with row and column labels, and then visually display the row and column positions of the steel stacks. The image before labeling is as follows: Figure 4 As shown, the labeled image is as follows Figure 5 As shown.

[0137] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for target detection and automatic sorting of steel long product stacks in images based on the RetinaNet algorithm, characterized in that, Includes the following steps: Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack position; Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model. Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates of each steel stack. Step (4): Based on the coordinates of the midpoint of the steel stack obtained in step (3), calculate the row and column positions of each steel stack. Step (5): Based on the calculation results in step (4), the steel stacks in the image are labeled with rows and columns, and then the row and column positions of the steel stacks are displayed visually.

2. The method for target detection and automatic sorting of steel long product images based on RetinaNet algorithm according to claim 1, characterized in that, In step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

3. The method for target detection and automatic sorting of steel long product images based on RetinaNet algorithm according to claim 1, characterized in that, In step (2), the ratio of training set to test set is 8:2 during training and testing.

4. The method for target detection and automatic sorting of steel long product images based on the RetinaNet algorithm according to claim 2, characterized in that, In step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows: The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive. The top left corner coordinate (x) of the first steel stack 11 y 11 The coordinates of the lower right corner are (x 12 y 12 ); The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 ); The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 ); …… Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack; Thus, the coordinates of the midpoints of all steel stacks are obtained as set X: X=(X1,X2,X3.......X n )。 5. The method for target detection and automatic sorting of steel long product images based on the RetinaNet algorithm according to claim 4, characterized in that, The specific method for step (4) is as follows: (1) Set the threshold β between rows: Where n is the total number of steel stack locations; the coordinate of the upper left corner of the (n-1)th steel stack location (x... (n-1)1 y (n-1)1 The coordinates of the lower right corner are (x (n-1)2 y (n-1)2 ); (2) Calculate the positional relationship between the coordinates of the midpoints of the stack to determine whether to divide into rows: Calculate the difference l between the y-coordinates of the midpoints of two adjacent steel stacks: …… Where l1 is the difference between the y-value of the midpoint of the second steel stack and the y-value of the midpoint of the first steel stack, and l2 is the difference between the y-value of the midpoint of the third steel stack and the y-value of the midpoint of the second steel stack. n-1 This is the difference between the y-value of the midpoint of the nth steel stack and the y-value of the midpoint of the (n-1)th steel stack. Then, a judgment is made: the difference in the y-coordinates of the midpoints of two adjacent steel stacks is greater than... Then, a line break is made between these two steel stacks, that is, the steel stack to which the minuend of the difference is calculated is moved to the next line, while the steel stack to which the subtrahend of the difference is calculated is kept in the current line; (3) Sort the x-values ​​of the midpoint coordinates of the steel stack in each row from smallest to largest, and then assign them a serial number; the serial number starts from 1 and is a positive integer; the serial number is the column number corresponding to the steel stack.

Images