Warehouse steel long material image automatic correction method combining RetinaNet and perspective transformation

Abstract

The invention relates to a method for automatically correcting images of long steel and iron in a warehouse in combination with RetinaNet and perspective transformation, and belongs to the technical field of machine vision. The method comprises the six steps of image acquisition and marking, construction of a target detection model, calculation of average midpoint coordinates, division of a midpoint coordinate set, calculation of correction coordinates and perspective transformation. According to the method, the steel stack position coordinates of the warehouse are detected and identified through the RetinaNet model, the steel stack position coordinates are extracted, the spatial relation of each steel stack position of the image is calculated, the optimal correction coordinate point is obtained, and perspective transformation is performed by using the correction coordinate point to obtain the correction image. According to the method, the problem of image inclination caused by the influence of light rays, steel placement, photographing positions and the like on acquired images is solved, and automatic correction of various inclined images in complex scenes can be met.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically relating to an automatic correction method for warehouse steel long product images that combines RetinaNet and perspective transformation. Background Technology

[0002] Traditional warehouse management suffers from numerous pain points in inventory counting, such as reliance on manual labor, miscounting, missed counts, high labor costs, and low operational efficiency. These problems not only increase operating costs but also impact a company's overall competitiveness and customer satisfaction. Existing WMS (Warehouse Management System) products still fall short in intelligent inventory counting, failing to fully meet companies' needs for intelligent inventory counting. During counting, image acquisition is affected by factors such as lighting (weather), the placement of steel materials (e.g., reversed placement), and the camera's position, leading to image tilt. Traditional image correction techniques, such as geometric transformations, histogram equalization, and filters, are typically based on fixed mathematical models or manually designed rules, requiring pre-assumptions of image distortion types (e.g., radial distortion, perspective distortion) or noise distribution. They also suffer from drawbacks such as insufficient multi-task processing, generalization ability, real-time performance, intelligence, and multimodal fusion, making effective inventory counting difficult.

[0003] Therefore, to address the complex image tilting problem of long steel products in warehouses, a practical and effective algorithm is needed to resolve the image tilt and achieve automatic correction. Overcoming the shortcomings of existing technologies is a pressing issue that needs to be addressed in this technical field. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies and provide an automatic correction method for warehouse steel long product images that combines RetinaNet and perspective transformation.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] An automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation includes the following steps:

[0007] Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location;

[0008] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0009] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates of each steel stack; then calculate the average midpoint coordinates of all steel stacks to obtain the average midpoint coordinates of all steel stacks.

[0010] Step (4): Determine the relative spatial relationship between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks to obtain the position of each steel stack on the entire steel stack image; the position includes the upper left corner, lower left corner, upper right corner and lower right corner; classify the midpoint coordinates of all steel stacks into four midpoint coordinate sets according to these four positions;

[0011] Step (5): Use the Euclidean distance algorithm to calculate the distance from the coordinate set of the midpoints of the top left, bottom left, top right, and bottom right corners to the average midpoint coordinates. Through the spatial relationship between distance and coordinates, obtain the optimal corrected coordinates of the top left, bottom left, top right, and bottom right corners.

[0012] Step (6): Using the optimal upper left, lower left, upper right, and lower right corner correction coordinates obtained in step (5), perform perspective transformation on the image of the steel stack to be identified to obtain the corrected image.

[0013] Furthermore, preferably, in step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

[0014] Furthermore, preferably, in step (2), the ratio of the training set to the test set is 8:2 during training and testing.

[0015] Furthermore, preferably, in step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows:

[0016] The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive.

[0017] The top left corner coordinate (x) of the first steel stack 11 y 11 The coordinates of the lower right corner are (x 12 y 12 );

[0018] The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 );

[0019] The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 );

[0020]

[0021] ...

[0022]

[0023] Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack;

[0024] Calculate the average coordinates of the midpoints of all steel stacks to obtain the average midpoint coordinates of all steel stacks; specifically, average the x-values ​​of the midpoint coordinates of all steel stacks to obtain the average x-value of the midpoint coordinates of all steel stacks; average the y-values ​​of the midpoint coordinates of all steel stacks to obtain the average y-value of the midpoint coordinates of all steel stacks.

[0025] Furthermore, preferably, the specific method of step (5) is as follows:

[0026] The Euclidean distance algorithm is used to calculate the Euclidean distance between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks in the four midpoint coordinate sets of the top left, bottom left, top right, and bottom right corners.

[0027] For each set of midpoint coordinates, the midpoint coordinates of the steel stack with the largest Euclidean distance from the average midpoint coordinates of all steel stacks are selected as the optimal correction coordinates for that set of midpoint coordinates.

[0028] Furthermore, preferably, the specific method of step (6) is as follows:

[0029] Let the optimal corrected coordinates for the top-left, bottom-left, top-right, and bottom-right corners be X1, X2, X3, and X4, respectively, specifically represented as X1(x1, y1), X2(x2, y2), X3(x3, y3), and X4(x4, y4). Assume these four optimal corrected coordinates X1, X2, X3, and X4 correspond to four points Q1, Q2, Q3, and Q4 of the corrected image.

[0030] The coordinates of Q1, Q2, Q3, and Q4 are specifically represented as Q1(u1, v1), Q2(u2, v2), Q3(u3, v3), and Q4(u4, v4);

[0031] Construct a system of linear equations:

[0032] Qi =H*X i i = 1, 2, 3, 4;

[0033] Substituting the coordinate values ​​and breaking them down yields:

[0034]

[0035] Rearrange into a system of linear equations, eliminating the denominators:

[0036]

[0037] Two equations are generated for each pair of corresponding points, and the solutions for the four pairs of corresponding points yield eight unknowns: the values ​​of a, b, c, d, e, f, g, and h.

[0038] Transforming the system of equations into matrix form and solving for the transformation matrix yields:

[0039]

[0040] The transformation matrix M is obtained by solving for the unknowns through matrix inversion:

[0041]

[0042] Then, the image is corrected using a perspective transformation matrix:

[0043] Perspective transformation satisfies:

[0044]

[0045] Among them, (x i y i ) are the coordinates of a point in the original image; (u i v i ) are the coordinates of a point in the transformed image.

[0046] Furthermore, preferably, it also includes:

[0047] Based on the coordinates of the midpoint of the steel stack obtained in the steps, calculate the row and column positions of each steel stack; then label the rows and columns of the steel stacks in the image, and then visually display the row and column positions of the steel stacks.

[0048] Furthermore, preferably, based on the coordinates of the midpoint of the steel stack obtained in the previous steps, the row and column positions of each steel stack are calculated. The specific method is as follows:

[0049] (1) Set the threshold β between rows:

[0050]

[0051] Where n is the total number of steel stack locations; the coordinate of the upper left corner of the (n-1)th steel stack location (x... (n-1)1 y (n-1)1 The coordinates of the lower right corner are (x (n-1)2 y (n-1)2 );

[0052] (2) Calculate the positional relationship between the coordinates of the midpoints of the stack to determine whether to divide into rows:

[0053] Calculate the difference l between the y-coordinates of the midpoints of two adjacent steel stacks:

[0054]

[0055] ...

[0056]

[0057] Where l1 is the difference between the y-value of the midpoint of the second steel stack and the y-value of the midpoint of the first steel stack, and l2 is the difference between the y-value of the midpoint of the third steel stack and the y-value of the midpoint of the second steel stack. n-1 This is the difference between the y-value of the midpoint of the nth steel stack and the y-value of the midpoint of the (n-1)th steel stack.

[0058] Then, a judgment is made: the difference in the y-coordinates of the midpoints of two adjacent steel stacks is greater than... Then, a line break is made between these two steel stacks, that is, the steel stack to which the minuend of the difference is calculated is moved to the next line, while the steel stack to which the subtrahend of the difference is calculated is kept in the current line;

[0059] (3) Sort the x-values ​​of the midpoint coordinates of the steel stack in each row from smallest to largest, and then assign them a serial number; the serial number starts from 1 and is a positive integer; the serial number is the column number corresponding to the steel stack.

[0060] In step (1) of the present invention, it is preferred to use a target detection and annotation tool to annotate the coordinates of each steel stack, but it is not limited to this and can also be annotated manually.

[0061] In step (1) of this invention, the same type of steel stacks to be identified means that the steel types are the same (e.g., they are all steel pipe stacks) and the stacking methods are the same.

[0062] In step (4) of this invention, the relative spatial relationship between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks is determined to obtain the position of each steel stack on the entire steel stack image; the position includes the upper left corner, lower left corner, upper right corner, and lower right corner; the midpoint coordinates of all steel stacks are classified into four midpoint coordinate sets according to these four positions; during classification, the average midpoint coordinates of all steel stacks are used as the center, and the horizontal and vertical lines passing through the center are used to divide the area into four parts, namely the upper left corner, lower left corner, upper right corner, lower right corner, upper left corner, lower right corner, upper left corner, lower right corner, upper left corner, lower right corner, upper right ... The coordinates of the midpoint of a steel stack are: bottom corner, top right corner, and bottom right corner. If the coordinates of the midpoint of a steel stack fall on the horizontal line to the right of the center, then the coordinates of the midpoint of the steel stack belong to the top right corner. If the coordinates of the midpoint of a steel stack fall on the horizontal line to the left of the center, then the coordinates of the midpoint of the steel stack belong to the bottom left corner. If the coordinates of the midpoint of a steel stack fall on the vertical line above the center, then the coordinates of the midpoint of the steel stack belong to the top left corner. If the coordinates of the midpoint of a steel stack fall on the vertical line below the center, then the coordinates of the midpoint of the steel stack belong to the bottom right corner.

[0063] The RetinaNet model used in this invention is an existing model.

[0064] This method can automatically and accurately correct images, support effective recognition of images of stacks of long steel products, meet the needs of complex scenarios, and contribute to the realization of intelligent warehouses.

[0065] Compared with the prior art, the beneficial effects of this invention are as follows:

[0066] Traditional image correction techniques, such as geometric transformations, histogram equalization, and filters, are typically based on fixed mathematical models or manually designed rules, requiring pre-assumptions of image distortion types, such as radial distortion, perspective distortion, or noise distribution. This invention addresses issues related to multi-task processing, generalization ability, real-time performance, intelligence, and multimodal fusion, enabling rapid application to the automatic correction of various tilted images in complex scenarios.

[0067] Compared with traditional image correction techniques, the method of this invention combines advanced visual AI target detection technology with traditional image correction technology. It has significant advantages in multi-task processing, generalization ability, real-time performance, intelligence, and multimodal fusion, and can meet the automatic correction of images acquired in various complex warehouse environments.

[0068] Traditional image correction requires manual adjustment of parameters and repair of details. This invention combines deep learning and traditional perspective transformation, eliminating the need for extensive manual adjustments to each image, saving labor costs, and quickly producing corrected images. This improves work efficiency, indirectly reduces time costs, enhances enterprise operational efficiency, and strengthens competitiveness. Attached Figure Description

[0069] Figure 1The flowchart obtained for the object detection model;

[0070] Figure 2 A 3D model diagram of the algorithm principle structure of the existing RetinaNet;

[0071] Figure 3 A 2D schematic diagram of the decomposition of the existing RetinaNet algorithm 3D model;

[0072] Figure 4 This is a diagram of the steel stacking before correction in an application example of the present invention;

[0073] Figure 5 This is a corrected diagram of steel stacking as an application example of the present invention. Detailed Implementation

[0074] The present invention will now be described in further detail with reference to the embodiments.

[0075] Those skilled in the art will understand that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention. Where specific techniques or conditions are not specified in the embodiments, they are performed in accordance with the techniques or conditions described in the literature in the field or according to the product instructions. Materials or equipment whose manufacturers are not specified are all conventional products that can be obtained by purchase.

[0076] Example 1

[0077] An automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation includes the following steps:

[0078] Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location;

[0079] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0080] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates of each steel stack; then calculate the average midpoint coordinates of all steel stacks to obtain the average midpoint coordinates of all steel stacks.

[0081] Step (4): Determine the relative spatial relationship between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks to obtain the position of each steel stack on the entire steel stack image; the position includes the upper left corner, lower left corner, upper right corner and lower right corner; classify the midpoint coordinates of all steel stacks into four midpoint coordinate sets according to these four positions;

[0082] Step (5): Use the Euclidean distance algorithm to calculate the distance from the coordinate set of the midpoints of the top left, bottom left, top right, and bottom right corners to the average midpoint coordinates. Through the spatial relationship between distance and coordinates, obtain the optimal corrected coordinates of the top left, bottom left, top right, and bottom right corners.

[0083] Step (6): Using the optimal upper left, lower left, upper right, and lower right corner correction coordinates obtained in step (5), perform perspective transformation on the image of the steel stack to be identified to obtain the corrected image.

[0084] Example 2

[0085] An automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation includes the following steps:

[0086] Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location;

[0087] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0088] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates of each steel stack; then calculate the average midpoint coordinates of all steel stacks to obtain the average midpoint coordinates of all steel stacks.

[0089] Step (4): Determine the relative spatial relationship between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks to obtain the position of each steel stack on the entire steel stack image; the position includes the upper left corner, lower left corner, upper right corner and lower right corner; classify the midpoint coordinates of all steel stacks into four midpoint coordinate sets according to these four positions;

[0090] Step (5): Use the Euclidean distance algorithm to calculate the distance from the coordinate set of the midpoints of the top left, bottom left, top right, and bottom right corners to the average midpoint coordinates. Through the spatial relationship between distance and coordinates, obtain the optimal corrected coordinates of the top left, bottom left, top right, and bottom right corners.

[0091] Step (6): Using the optimal upper left, lower left, upper right, and lower right corner correction coordinates obtained in step (5), perform perspective transformation on the image of the steel stack to be identified to obtain the corrected image.

[0092] In step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

[0093] In step (2), the ratio of training set to test set samples is 8:2 during training and testing.

[0094] In step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows:

[0095] The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive.

[0096] The top left corner coordinate (x) of the first steel stack 11 y 11 The coordinates of the lower right corner are (x 12 y 12 );

[0097] The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 );

[0098] The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 );

[0099]

[0100] ...

[0101]

[0102] Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack;

[0103] Calculate the average coordinates of the midpoints of all steel stacks to obtain the average midpoint coordinates of all steel stacks; specifically, average the x-values ​​of the midpoint coordinates of all steel stacks to obtain the average x-value of the midpoint coordinates of all steel stacks; average the y-values ​​of the midpoint coordinates of all steel stacks to obtain the average y-value of the midpoint coordinates of all steel stacks.

[0104] The specific method for step (5) is as follows:

[0105] The Euclidean distance algorithm is used to calculate the Euclidean distance between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks in the four midpoint coordinate sets of the top left, bottom left, top right, and bottom right corners.

[0106] For each set of midpoint coordinates, the midpoint coordinates of the steel stack with the largest Euclidean distance from the average midpoint coordinates of all steel stacks are selected as the optimal correction coordinates for that set of midpoint coordinates.

[0107] The characteristic feature is that the specific method of step (6) is as follows:

[0108] Let the optimal corrected coordinates for the top-left, bottom-left, top-right, and bottom-right corners be X1, X2, X3, and X4, respectively, specifically represented as X1(x1, y1), X2(x2, y2), X3(x3, y3), and X4(x4, y4). Assume these four optimal corrected coordinates X1, X2, X3, and X4 correspond to four points Q1, Q2, Q3, and Q4 of the corrected image.

[0109] The coordinates of Q1, Q2, Q3, and Q4 are specifically represented as Q1(u1, v1), Q2(u2, v2), Q3(u3, v3), and Q4(u4, v4);

[0110] Construct a system of linear equations:

[0111] Q i =H*X i i = 1, 2, 3, 4;

[0112] Substituting the coordinate values ​​and breaking them down yields:

[0113]

[0114] Rearrange into a system of linear equations, eliminating the denominators:

[0115]

[0116] Two equations are generated for each pair of corresponding points, and the solutions for the four pairs of corresponding points yield eight unknowns: the values ​​of a, b, c, d, e, f, g, and h.

[0117] Transforming the system of equations into matrix form and solving for the transformation matrix yields:

[0118]

[0119] The transformation matrix M is obtained by solving for the unknowns through matrix inversion:

[0120]

[0121] Then, the image is corrected using a perspective transformation matrix:

[0122] Perspective transformation satisfies:

[0123]

[0124] Among them, (x i y i ) are the coordinates of a point in the original image; (u i ,v i ) are the coordinates of a point in the transformed image.

[0125] Example 3

[0126] An automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation includes the following steps:

[0127] Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location;

[0128] Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model.

[0129] Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates of each steel stack; then calculate the average midpoint coordinates of all steel stacks to obtain the average midpoint coordinates of all steel stacks.

[0130] Step (4): Determine the relative spatial relationship between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks to obtain the position of each steel stack on the entire steel stack image; the position includes the upper left corner, lower left corner, upper right corner and lower right corner; classify the midpoint coordinates of all steel stacks into four midpoint coordinate sets according to these four positions;

[0131] Step (5): Use the Euclidean distance algorithm to calculate the distance from the coordinate set of the midpoints of the top left, bottom left, top right, and bottom right corners to the average midpoint coordinates. Through the spatial relationship between distance and coordinates, obtain the optimal corrected coordinates of the top left, bottom left, top right, and bottom right corners.

[0132] Step (6): Using the optimal upper left, lower left, upper right, and lower right corner correction coordinates obtained in step (5), perform perspective transformation on the image of the steel stack to be identified to obtain the corrected image.

[0133] In step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

[0134] In step (2), the ratio of training set to test set samples is 8:2 during training and testing.

[0135] In step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows:

[0136] The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive.

[0137] The top left corner coordinate (x) of the first steel stack 11 y 11 The coordinates of the lower right corner are (x 12 y 12 );

[0138] The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 );

[0139] The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 );

[0140]

[0141] ...

[0142]

[0143] Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack;

[0144] Calculate the average coordinates of the midpoints of all steel stacks to obtain the average midpoint coordinates of all steel stacks; specifically, average the x-values ​​of the midpoint coordinates of all steel stacks to obtain the average x-value of the midpoint coordinates of all steel stacks; average the y-values ​​of the midpoint coordinates of all steel stacks to obtain the average y-value of the midpoint coordinates of all steel stacks.

[0145] The specific method for step (5) is as follows:

[0146] The Euclidean distance algorithm is used to calculate the Euclidean distance between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks in the four midpoint coordinate sets of the top left, bottom left, top right, and bottom right corners.

[0147] For each set of midpoint coordinates, the midpoint coordinates of the steel stack with the largest Euclidean distance from the average midpoint coordinates of all steel stacks are selected as the optimal correction coordinates for that set of midpoint coordinates.

[0148] The characteristic feature is that the specific method of step (6) is as follows:

[0149] Let the optimal corrected coordinates for the top-left, bottom-left, top-right, and bottom-right corners be X1, X2, X3, and X4, respectively, specifically represented as X1(x1, y1), X2(x2, y2), X3(x3, y3), and X4(x4, y4). Assume these four optimal corrected coordinates X1, X2, X3, and X4 correspond to four points Q1, Q2, Q3, and Q4 of the corrected image.

[0150] The coordinates of Q1, Q2, Q3, and Q4 are specifically represented as Q1(u1, v1), Q2(u2, v2), Q3(u3, v3), and Q4(u4, v4);

[0151] Construct a system of linear equations:

[0152] Q i =H*X i i = 1, 2, 3, 4;

[0153] Substituting the coordinate values ​​and breaking them down yields:

[0154]

[0155] Rearrange into a system of linear equations, eliminating the denominators:

[0156]

[0157] Two equations are generated for each pair of corresponding points, and the solutions for the four pairs of corresponding points yield eight unknowns: the values ​​of a, b, c, d, e, f, g, and h.

[0158] Transforming the system of equations into matrix form and solving for the transformation matrix yields:

[0159]

[0160] The transformation matrix M is obtained by solving for the unknowns through matrix inversion:

[0161]

[0162] Then, the image is corrected using a perspective transformation matrix:

[0163] Perspective transformation satisfies:

[0164]

[0165] Among them, (x i y i ) are the coordinates of a point in the original image; (u i v i ) are the coordinates of a point in the transformed image.

[0166] Also includes:

[0167] Based on the coordinates of the midpoint of the steel stack obtained in the steps, calculate the row and column positions of each steel stack; then label the rows and columns of the steel stacks in the image, and then visually display the row and column positions of the steel stacks.

[0168] Based on the coordinates of the midpoint of the steel stack obtained in the steps, calculate the row and column positions of each steel stack. The specific method is as follows:

[0169] (1) Set the threshold β between rows:

[0170]

[0171] Where n is the total number of steel stack locations; the coordinate of the upper left corner of the (n-1)th steel stack location (x... (n-1)1 y (n-1)1 The coordinates of the lower right corner are (x (n-1)2 y (n-1)2 );

[0172] (2) Calculate the positional relationship between the coordinates of the midpoints of the stack to determine whether to divide into rows:

[0173] Calculate the difference l between the y-coordinates of the midpoints of two adjacent steel stacks:

[0174]

[0175] ...

[0176]

[0177] Where l1 is the difference between the y-value of the midpoint of the second steel stack and the y-value of the midpoint of the first steel stack, and l2 is the difference between the y-value of the midpoint of the third steel stack and the y-value of the midpoint of the second steel stack. n-1 This is the difference between the y-value of the midpoint of the nth steel stack and the y-value of the midpoint of the (n-1)th steel stack.

[0178] Then, a judgment is made: the difference in the y-coordinates of the midpoints of two adjacent steel stacks is greater than... Then, a line break is made between these two steel stacks, that is, the steel stack to which the minuend of the difference is calculated is moved to the next line, while the steel stack to which the subtrahend of the difference is calculated is kept in the current line;

[0179] (3) Sort the x-values ​​of the midpoint coordinates of the steel stack in each row from smallest to largest, and then assign them a serial number; the serial number starts from 1 and is a positive integer; the serial number is the column number corresponding to the steel stack.

[0180] Application Examples

[0181] An automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation, characterized by the following steps:

[0182] 1) Collect images of the warehouse's long steel products inventory using an image acquisition device and upload them to the system, using a sufficient number of images as the training dataset.

[0183] 2) Traverse all images and use object detection annotation tools to annotate the steel stacks in the sample dataset to obtain the corresponding label files. The label files and the original images together form the image dataset. Randomly select 80% of the sample set as the training set and 20% as the test set.

[0184] 3) Train the RetinaNet model using the training dataset, and test the trained RetinaNet model using the test set to obtain the object detection model (the process is as follows). Figure 1 As shown, the principle is as follows Figure 2 and Figure 3 (As shown).

[0185] 4) Acquire images of the steel stacks to be identified in real time and input them into the target detection model obtained in step 2) to obtain the coordinates of each steel stack position; then use the obtained coordinates of each steel stack position to calculate the midpoint coordinates of each steel stack position, and then calculate the average midpoint coordinates of all steel stack positions to obtain the average midpoint coordinates of all steel stack positions.

[0186] 5) Determine the relative spatial relationship between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks to obtain the position of each steel stack on the entire steel stack image; the position includes the upper left corner, lower left corner, upper right corner, and lower right corner; classify the midpoint coordinates of all steel stacks into four midpoint coordinate sets according to these four positions;

[0187] 6) Use the Euclidean distance algorithm to calculate the distance from the coordinate set of the midpoints of the top left, bottom left, top right, and bottom right corners to the average midpoint coordinates. Through the spatial relationship between distance and coordinates, obtain the optimal corrected coordinates of the top left, bottom left, top right, and bottom right corners.

[0188] 7) Using the optimal corrected coordinates of the top left, bottom left, top right, and bottom right corners obtained in step 6), perform a perspective transformation on the image of the steel stack to be identified to obtain the corrected image. (The image before perspective transformation and correction is shown below.) Figure 4 As shown, the corrected image is as follows Figure 5 (As shown).

[0189] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation, characterized in that, Includes the following steps: Step (1): Collect images of steel stacks in steel warehouses of the same type as the steel stacks to be identified, and annotate them, marking the coordinates of each steel stack location; Step (2): The RetinaNet model is trained and tested using the steel stack image collected in step (1) as input and the corresponding labeled steel stack coordinates obtained in step (1) as output, to obtain the target detection model. Step (3): Real-time acquisition of images of the steel stacks to be identified, input into the target detection model obtained in step (2), and obtain the stack coordinates of each steel stack; then, using the obtained stack coordinates of each steel stack, calculate the midpoint coordinates of each steel stack; then calculate the average midpoint coordinates of all steel stacks to obtain the average midpoint coordinates of all steel stacks. Step (4): Determine the relative spatial relationship between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks to obtain the position of each steel stack on the entire steel stack image; the position includes the upper left corner, lower left corner, upper right corner and lower right corner; classify the midpoint coordinates of all steel stacks into four midpoint coordinate sets according to these four positions; Step (5): Use the Euclidean distance algorithm to calculate the distance from the coordinate set of the midpoints of the top left, bottom left, top right, and bottom right corners to the average midpoint coordinates. Through the spatial relationship between distance and coordinates, obtain the optimal corrected coordinates of the top left, bottom left, top right, and bottom right corners. Step (6): Using the optimal upper left, lower left, upper right, and lower right corner correction coordinates obtained in step (5), perform perspective transformation on the image of the steel stack to be identified to obtain the corrected image.

2. The automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation according to claim 1, characterized in that, In step (1), an image acquisition device is used to acquire images of steel stacks in the steel warehouse; the coordinates of each steel stack position are marked, including the coordinates of the upper left corner and the lower right corner.

3. The automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation according to claim 1, characterized in that, In step (2), the ratio of training set to test set samples is 8:2 during training and testing.

4. The automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation according to claim 1, characterized in that, In step (3), the specific method for calculating the midpoint coordinates of each steel stack is as follows: The coordinate system for the steel stack is based on the origin at the top left corner of the image, with the X-axis to the right, where the values ​​increase as you move to the right and are all positive; and the Y-axis to the bottom, where the values ​​increase as you move down and are all positive. The top left corner coordinate (x) of the first steel stack 11 y 11 The coordinates of the lower right corner are (x 12 y 12 ); The top left corner coordinate (x) of the second steel stack 21 y 21 The coordinates of the lower right corner are (x 22 y 22 ); The coordinates of the top left corner of the nth steel stack (x n1 y n1 The coordinates of the lower right corner are (x n2 y n2 ); …… Where: X1 represents the coordinates of the midpoint of the first steel stack, X2 represents the coordinates of the midpoint of the second steel stack, X... n Represents the coordinates of the midpoint of the nth steel stack; Calculate the average coordinates of the midpoints of all steel stacks to obtain the average midpoint coordinates of all steel stacks; specifically, average the x-values ​​of the midpoint coordinates of all steel stacks to obtain the average x-value of the midpoint coordinates of all steel stacks; average the y-values ​​of the midpoint coordinates of all steel stacks to obtain the average y-value of the midpoint coordinates of all steel stacks.

5. The automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation according to claim 4, characterized in that, The specific method for step (5) is as follows: The Euclidean distance algorithm is used to calculate the Euclidean distance between the midpoint coordinates of each steel stack and the average midpoint coordinates of all steel stacks in the four midpoint coordinate sets of the top left, bottom left, top right, and bottom right corners. For each set of midpoint coordinates, the midpoint coordinates of the steel stack with the largest Euclidean distance from the average midpoint coordinates of all steel stacks are selected as the optimal correction coordinates for that set of midpoint coordinates.

6. The automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation according to claim 1, characterized in that, The specific method for step (6) is as follows: Let the optimal corrected coordinates for the top-left, bottom-left, top-right, and bottom-right corners be X1, X2, X3, and X4, respectively, specifically represented as X1(x1, y1), X2(x2, y2), X3(x3, y3), and X4(x4, y4). Assume these four optimal corrected coordinates X1, X2, X3, and X4 correspond to four points Q1, Q2, Q3, and Q4 of the corrected image. The coordinates of Q1, Q2, Q3, and Q4 are specifically represented as Q1(u1, v1), Q2(u2, v2), Q3(u3, v3), and Q4(u4, v4); Construct a system of linear equations: Q i =H*X i ;i=1、2、3、4; Substituting the coordinate values ​​and breaking them down yields: Rearrange into a system of linear equations, eliminating the denominators: Two equations are generated for each pair of corresponding points, and the solutions for the four pairs of corresponding points yield eight unknowns: the values ​​of a, b, c, d, e, f, g, and h. Transforming the system of equations into matrix form and solving for the transformation matrix yields: The transformation matrix M is obtained by solving for the unknowns through matrix inversion: Then, the image is corrected using a perspective transformation matrix: Perspective transformation satisfies: Among them, (x i y i () represents the coordinates of a point in the original image; (ui,vi) are the coordinates of a point in the transformed image.

7. The automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation according to claim 1, characterized in that, Also includes: Based on the coordinates of the midpoint of the steel stack obtained in the steps, calculate the row and column positions of each steel stack; then label the rows and columns of the steel stacks in the image, and then visually display the row and column positions of the steel stacks.

8. The automatic correction method for warehouse steel long product images combining RetinaNet and perspective transformation according to claim 7, characterized in that, Based on the coordinates of the midpoint of the steel stack obtained in the steps, calculate the row and column positions of each steel stack. The specific method is as follows: (1) Set the threshold β between rows: Where n is the total number of steel stack locations; the coordinate of the upper left corner of the (n-1)th steel stack location (x... (n-1)1 y (n-1)1 The coordinates of the lower right corner are (x (n-1)2 y (n-1)2 ); (2) Calculate the positional relationship between the coordinates of the midpoints of the stack to determine whether to divide into rows: Calculate the difference l between the y-coordinates of the midpoints of two adjacent steel stacks: …… Where l1 is the difference between the y-value of the midpoint of the second steel stack and the y-value of the midpoint of the first steel stack, and l2 is the difference between the y-value of the midpoint of the third steel stack and the y-value of the midpoint of the second steel stack. n-1 This is the difference between the y-value of the midpoint of the nth steel stack and the y-value of the midpoint of the (n-1)th steel stack. Then, a judgment is made: the difference in the y-coordinates of the midpoints of two adjacent steel stacks is greater than... Then, a line break is made between these two steel stacks, that is, the steel stack to which the minuend of the difference is calculated is moved to the next line, while the steel stack to which the subtrahend of the difference is calculated is kept in the current line; (3) Sort the x-values ​​of the midpoint coordinates of the steel stack in each row from smallest to largest, and then assign them a serial number; the serial number starts from 1 and is a positive integer; the serial number is the column number corresponding to the steel stack.

Images