An automatic control method for vehicle-mounted slab entry and exit

Abstract

This invention discloses an automatic control method for the entry and exit of vehicle-mounted slabs, relating to the field of automatic control of vehicle-mounted slab entry and exit in hot rolling production of steel enterprises. The method first uses a camera to capture images of the vehicle-mounted slabs, converts them into bird's-eye views, and then creates a dataset for training an image segmentation neural network. The mIoU accuracy of the trained segmentation model is approximately 0.97. The method then uses a camera to capture images of the vehicle-mounted slabs, converts them into bird's-eye views, and feeds them into the image segmentation model. The centroid coordinates of the truck or slab portion of the image are calculated from the segmentation results, and the actual coordinate values ​​of the truck or slab are calculated and fed back to the crane system, completing the intelligent entry and exit of the vehicle-mounted slabs. Finally, this method was put into use at the Shagang plant. After actual testing, it was found that after using this method, the crane clamps can operate effectively based on the positioning results, replacing manual control of the crane positioning and realizing intelligent entry and exit of vehicle-mounted slabs.

Description

Technical Field

[0001] This invention relates to the fields of metallurgical machinery and automation, and artificial intelligence, specifically to a control method for automatically loading and unloading slabs from and out of a warehouse by calculating the positioning coordinates of the slabs. Background Technology

[0002] The transportation of hot-rolled slabs is a crucial link in the production line of steel enterprises. Transporting the finished slabs out of or into the warehouse by truck is a vital step in slab transportation within the slab storage area. Currently, the automation level of the hot-rolled slab storage process is low. Many steel enterprises still rely on manual labor in three shifts, 24 hours a day. Truck drivers park their trucks in the storage area, and workers operate overhead cranes to clamp the slabs. The entire process requires ground personnel to confirm accuracy to ensure operational safety. This process is not only labor-intensive and inefficient but also poses safety hazards. Furthermore, it is highly susceptible to human error; prolonged operation can lead to improper operation by crane operators, potentially damaging the slabs or trucks and causing economic losses to the plant. Summary of the Invention

[0003] The purpose of this invention is to provide a method for calculating the positioning coordinates of vehicle-mounted slabs. Specifically, it involves image segmentation and recognition of the automatic entry and exit of vehicle-mounted slabs in a slab warehouse, using OpenCV image processing algorithms to find the centroid coordinates of the segmentation results, and converting the centroid coordinates in the segmented image into the actual coordinates of the vehicle through a coordinate transformation formula. This is then fed back to the vehicle control and scheduling system, thereby realizing the automatic entry and exit of vehicle-mounted slabs in the slab warehouse area of ​​the steel plant, accelerating the loading pace, and improving the transportation efficiency of the steel plant workshop.

[0004] To achieve the above objectives, the present invention provides an automatic control method for the entry and exit of vehicle-mounted slabs, comprising the following steps:

[0005] Acquire video recordings of slabs being moved in and out of the warehouse on-site, and extract each frame of the video recordings to obtain images of the slabs being transported by trucks.

[0006] Each image of a vehicle-mounted slab entering and leaving the warehouse is converted into a bird's-eye view and labeled to create the dataset needed to train the vehicle-mounted slab recognition model.

[0007] Training an image segmentation neural network model;

[0008] A coordinate recognition model was deployed on-site, and the centroid coordinates of the segmented image results were obtained by querying the algorithm.

[0009] The coordinates of the centroid of the truck or the slab on board are converted into the actual coordinates of the vehicle and fed back to the vehicle dispatching system to complete the automatic entry and exit control of the slab on board.

[0010] More specifically, the automatic control of the vehicle-mounted slabs entering and leaving the warehouse is as follows:

[0011] When slabs are being shipped out of the warehouse, the empty trucks stop within the marked rectangular parking lines in the warehouse area. The truck driver swipes his card, triggering an identification signal. The crane then picks up the slabs from the warehouse stack and transfers them to the truck for shipment. When slabs are being shipped into the warehouse, the truck-mounted slabs stop within the marked parking lines in the warehouse area. The truck driver swipes his card, triggering an identification signal. The crane then picks up the slabs from the truck and transfers them to the warehouse stack, completing the shipment process.

[0012] The recognition program repeatedly accesses the database. After the truck driver swipes his card to obtain the recognition signal, it performs image segmentation and recognition, and calculates coordinates.

[0013] First, capture a frame of the original image from the camera, with a size of 2560x1440. Select the coordinates of the four vertices of the parking space in the image, calculate the transparency matrix of the image using cv2.getPerspectiveTransform() in opecv, and then use cv2.warpPerspective() to obtain the bird's-eye view of the image. Then, feed it into the image segmentation network model to obtain the segmentation results of the truck or the slab. Calculate the centroid coordinates of the mask obtained from the image segmentation network, and use cv2.moments() to obtain the centroid coordinates of the truck or the slab.

[0014] Beforehand, test the correct coordinates of 3-6 sets of slab clamps from left to right, recording the image recognition coordinates each time. Then, compare the two, establish an equation, solve for the image-to-crane conversion formula, and use this formula to convert the image coordinates into the actual coordinates of the crane. The coordinates are then transmitted to the crane's control and scheduling system through the database to complete the slab entry and exit operations.

[0015] The image segmentation network model is trained using the following method:

[0016] Cameras were installed in the slab warehouse area to record the daily entry and exit of slabs from vehicles.

[0017] The video recording of the daily operation of the vehicle-mounted slab is captured and saved locally, and then each frame is extracted from it.

[0018] Batch process all images, deleting those without trucks or those obscured by vehicles.

[0019] Convert each image of the vehicle-mounted slab entering and leaving the warehouse into a bird's-eye view.

[0020] The image annotation tool labelme was used to segment and annotate the truck and slab sections in the obtained bird's-eye view image.

[0021] Delete images that are blurry or were captured with significant shaky footage during the annotation process;

[0022] Convert the JSON file obtained from the labeled image segmentation results into a mask.

[0023] 6584 original images and their corresponding mask images were selected as the training set for the model, and the remaining 471 original images and their corresponding mask images were selected as the validation set. The number of images collected and the division ratio were not fixed and were determined according to the on-site conditions.

[0024] By employing a segmentation neural network model, setting hyperparameters, using data augmentation, and after adjustments and optimizations, a segmentation network recognition model was obtained through training.

[0025] The present invention has the following advantages over the prior art:

[0026] The process involves truck drivers stopping and swiping their cards, using an image segmentation neural network to segment and process the bird's-eye view captured by the camera, then using image processing algorithms in OpenCV to obtain the centroid coordinates of the segmented result, and finally converting them into the actual coordinates of the vehicle using a pre-measured and calculated coordinate transformation formula. This is then fed back to the vehicle's dispatch and control system, completing the automatic entry and exit process of truck-mounted slabs in the steel plant's slab warehouse area.

[0027] This invention not only reduces manual labor while ensuring accuracy meets on-site requirements, but also shortens the time spent in the slab warehouse entry and exit process. While avoiding some safety hazards, it also enables 24-hour uninterrupted operation, improving the overall transportation efficiency of the production workshop. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the site layout involved in the present invention;

[0029] Figure 2 This is a flowchart of an automatic control method for the entry and exit of vehicle-mounted slab blanks provided by the present invention;

[0030] Figure 3 This is a schematic diagram of the vehicle-mounted slab positioning coordinate calculation method used in this invention; Detailed Implementation

[0031] To more conveniently illustrate the specific process of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is worth noting that the specific embodiments described herein are for explanation and illustration only, and are not intended to limit the invention.

[0032] Embodiments of the present invention provide an automatic control method for the entry and exit of vehicle-mounted slabs, such as... Figure 1 The diagram shown is a schematic representation of the site layout.

[0033] There are two operating methods for truck-mounted slabs in the slab warehouse: outbound and inbound. When the slab warehouse needs to move slabs from their designated locations according to plan, the truck driver drives the truck into the slab warehouse area and parks it in the marked parking area (points A, B, C, D in the diagram). The driver then swipes their card at the designated location to trigger the recognition signal. It is important to note that the driver needs to be aware of the location where the truck may be operating and ensure that the truck is not near the camera when swiping the card, so as not to obstruct the camera or cause camera shake, resulting in incorrect recognition calculations. After the identification and calculation process is completed, the crane will retrieve the slab from the corresponding stack in the warehouse according to the outbound plan. Then, based on the coordinates obtained from the identification and calculation, the slab will be placed onto the truck. Once the placement is correct, the crane will leave, and the truck driver will drive away from the warehouse to complete the outbound operation. Conversely, when slabs need to be put into storage according to the plan, the truck driver will park the truck carrying the slabs, swipe a card to trigger the identification and calculation signal, and the crane will retrieve the slab from the parking line according to the coordinates obtained from the identification and calculation. Then, it will place the slab in the corresponding position according to the storage plan to complete the storage operation.

[0034] like Figure 2 The flowchart of the automatic control method for vehicle-mounted slab entry and exit from the warehouse of the present invention is shown below:

[0035] If it is slabs leaving the warehouse, the truck driver drives the empty truck into the warehouse area, parks within the marked parking lines, observes the driving position, and then swipes the card to trigger the recognition signal. If it is slabs entering the warehouse, the truck driver needs to drive the truck carrying the slabs into the warehouse area, park the truck, observe the driving position, and then swipe the card to trigger the recognition signal.

[0036] The recognition program continuously accesses the recognition signal in the database, defaulting to 0 and performing no recognition. The signal changes to 1 after the truck driver swipes their card, triggering the recognition. A single frame of the original image is captured from the video stream of the on-site camera using the camera's IP address, username, and password. Due to complex on-site conditions and occasional network fluctuations, a check is added to determine if the camera has successfully acquired the original image. If successful, the recognition process continues; if it fails, it continues trying. After three consecutive failures, the program reports the cause to the database, providing an error report and awaiting intervention from on-site workers to troubleshoot the network.

[0037] Due to limitations of the site conditions, the camera's installation position and angle are not a complete top-down view. Therefore, to facilitate subsequent coordinate calculations, the original image acquired by the video needs to be converted into a bird's-eye view similar to a top view. The original image size is 2560x1440. Based on the location of the parking lines on site, the coordinates of the four vertices of the bird's-eye view are marked as A(1030, 302), B(1609, 298), C(813, 1278), and D(1847, 1291). The coordinates of these four points are not fixed and can be adjusted. The driving coordinates are as follows: A... t(121365,12220), B t (131825,12220), C t (121365,33980), D t The coordinates (131825, 33980) were both manually measured and labeled, and therefore contain some error. The perspective matrix M was calculated using OpenCV's built-in `cv2.getPerspectiveTransform()` and the coordinates of the four calibrated vertices. The final bird's-eye view was then obtained using `cv2.warpPerspective()` and the defined bird's-eye view size of M relative to the image.

[0038] To process the aforementioned bird's-eye view, a custom image segmentation neural network was trained. First, a dataset was created, collecting images from the daily inbound and outbound operations of the warehouse area, and selecting high-quality images that were not obstructed by vehicles or exhibited motion blur. Labelme was used for annotation, and the results were converted into a mask. The number of images for the training and validation sets was selected based on the scale of the collected images. The custom segmentation neural network model was then trained using DeepLabv3Plus, with an input image size of 512x512. Data augmentation techniques such as RandomCrop, RandomFlip, and Pad were employed. The training consisted of 300 epochs, using SGD with a learning rate of 0.01, a momentum of 0.9, and a weight decay of 0.0005. The final mIoU value was 0.97. Using this model to segment the bird's-eye view, the resulting mask showed that the truck portion had a pixel value of 110, the slab portion had a pixel value of 55, and the remaining portions had a pixel value of 0.

[0039] Calculate the centroid coordinates of the truck or slab portion in the mask. Use the cv2.moments() image processing algorithm in OpenCV to obtain the centroid coordinates of the truck or slab in the image. If only the truck coordinates are available, it is considered an outbound task; if the slab coordinates are available, it is considered an inbound task.

[0040] The centroid coordinates in the mask above are coordinates from the image. To ensure the vehicle correctly navigates to its designated location, these coordinates need to be converted to the vehicle's actual coordinates. Firstly, based on the principle of proportionality, the bird's-eye view of the image corresponds to the area within the parking lines on the ground. Therefore, the proportion of the centroid coordinates in the image is equal to the proportion of the centroid of the truck or slab on the actual ground within the parking lines. Based on this principle of proportionality, the vehicle's identification coordinates can be initially calculated through this conversion.

[0041] X true =A x-true+(B x-true -A x-true )÷img w ×x center ...(1)

[0042] Where X true The actual driving coordinates are calculated proportionally, A x-true and B x-true It is driving Figure 1 The actual x-coordinates of points A and B on the driving axis, img w It is the width of the image, x center It is the x-axis coordinate of the centroid of the image recognition result.

[0043] Based on preliminary on-site investigation and testing, it was found that the camera exhibits barrel-shaped deformation. This would cause proportional calculations to fail to meet the requirements of on-site operations, so an error correction formula needs to be added. On-site testing from left to right yielded 3-6 sets of values. This invention uses 3 sets as an example: the left set x1, the middle set x2, and the right set x3 represent the correct clamping coordinates of the clamps during manual operation of the crane movement. 11 x 22 x 33 These are the coordinate formulas for image recognition calculations during each vehicle journey, and then the corresponding (x1-x) formulas are calculated. 11 (x2-x) 22 (x3-x) 33 Three sets of deviation values ​​can be used to approximate the solution of the deviation equation:

[0044] D x = a×x+b...(2)

[0045] These results can also be fitted using quadratic equations based on the different distribution patterns of the deviation points:

[0046] D x =a×x 2 +bx+c...(3)

[0047] In (2) and (3), D is the deviation between the actual test coordinate value of the vehicle and the image recognition value, x is the centroid coordinate of the image segmentation result, and a, b, and c are constants obtained by measurement and calculation.

[0048] Finally, substituting the deviation value into (1) yields the final result:

[0049] X true =A true +(B true -A true )÷img width ×x center +Dx ...(4)

[0050] Similarly, the formula for calculating the true coordinates of vehicle recognition in the y-direction can be obtained as follows:

[0051] Y true =A y-true +(C y-true -A y-true )÷img h ×y center +D y ...(5)

[0052] Where Y true The true coordinate values ​​calculated for the slab's crane identification, A y-true and C y-true yes Figure 1 The actual y-axis coordinates of points A and C during driving, img h It is the height of the image, x center The y-axis coordinates of the centroid of the image recognition result, D y It is the calculated value of the y-axis deviation formula, and it is a constant in this invention.

[0053] The formula for calculating the coordinates of trucks is the same as that for slabs; simply modify the (x, y) variable name.

[0054] After substituting the centroid coordinates of the mask image obtained from the segmentation model into the actual coordinate values, the results are fed back to the dispatching and control system of the train through the database. Based on the coordinates, the train is controlled to move to the designated location to perform the operation, completing the outbound or inbound process of the warehouse.

[0055] Figure 3 An example diagram of the method for calculating the positioning coordinates of the vehicle-mounted slab is provided:

[0056] First, the truck driver drives the truck to the warehouse area, completes the parking at the designated location and swipes the card to observe the driving position, triggering the database recognition information, changing the recognition flag from 0 to 1, and the recognition program cyclically accesses the recognition flag signal in the database. If it is 1, it will collect a frame image from the camera. According to the four-point coordinates and parking line coordinates drawn, the image is converted into a bird's-eye view with a size of 1033x1039. After converting the image into a bird's-eye view, it is sent into the segmentation model to obtain the segmentation result. The black part is the background image with a pixel value of 0, and the light white part is the slab part with a pixel value of 55. The light white part of the segmentation is processed by cv2.moments() to obtain the centroid coordinates of the slab as (476, 506). These coordinates are the coordinates in the image and need to be converted into the actual coordinates of the driving vehicle through the coordinate transformation formula. After substituting into the calculation formulas of (4) and (5), X is calculated. true The value is 126353, Y trueThe result is 25917. Finally, the result (X) true Y true The data is input into the database, and the identification flag in the database is modified to 0, awaiting the next identification. After the on-site crane obtains the identification result, it calls the control system to verify the result. If it is an outbound plan, the corresponding truck coordinates are used; if it is an inbound plan, the corresponding slab coordinates are used. In this case, it is an inbound plan. After a simple verification, the crane is dispatched to the corresponding location to pick up the slab and place it in the planned slab warehouse stack. After the truck driver waits for the crane to safely pick up the slab from the truck and remove it, the truck can leave the warehouse area.

[0057] The above description is a preferred embodiment of the present invention. Those skilled in the art can understand and implement the present invention, and some adjustments and improvements can be made without departing from the principle scope of the present invention. These adjustments and improvements should also be considered within the protection scope of the present invention.

Claims

1. An automatic control method for the entry and exit of vehicle-mounted slabs, characterized in that, Includes the following steps: Acquire video recordings of slabs being moved in and out of the warehouse on-site, and extract each frame of the video recordings to obtain images of the slabs being transported by trucks. Each image of a vehicle-mounted slab entering and leaving the warehouse is converted into a bird's-eye view and labeled to create the dataset needed to train the vehicle-mounted slab recognition model. Training an image segmentation neural network model; A coordinate recognition model was deployed on-site, and the centroid coordinates of the segmented image results were obtained by querying the algorithm. The coordinates of the centroid of the truck or the slab on board are converted into the actual coordinates of the vehicle and fed back to the vehicle dispatching system to complete the automatic entry and exit control of the slab on board. The on-site deployment of the coordinate recognition model uses an algorithm to query the centroid coordinates of the segmented image results. The specific method is as follows: After the driver parks the truck in the parking line marked in the warehouse area, he swipes the automatic operation card at the entrance of the warehouse area; the database is accessed cyclically to obtain the recognition signal after the driver swipes the card, and recognition begins; a frame of image from the camera is captured, processed into a bird's-eye view, and segmented to separate the truck or the slab on the truck in the image, and the centroid coordinates are found using the image processing algorithm in OpenCV. The automatic entry and exit control method for truck-mounted slabs based on the centroid coordinates of the truck or slab is as follows: Measure the driving coordinates of the four vertices of the rectangular parking line drawn in the exit area; calculate the conversion formula between the two using multiple sets of real and correct coordinates of manually scheduled trucks entering and exiting the warehouse and the corresponding program-recognized image coordinates; convert the image coordinates of this segmentation and recognition result into driving coordinates according to the pre-tested and calculated coordinate conversion formula, and input them into the control program through the database to control the truck to perform operations, thus completing the automatic process of truck entry and exit from the warehouse. The calculation method of the coordinate transformation formula is as follows: First, test the correct coordinates of 3-6 sets of crane clamp slabs from left to right, record the image recognition coordinates each time, then compare the two, establish the equation, and solve to obtain the image to crane formula.

2. The automatic control method for the entry and exit of vehicle-mounted slabs according to claim 1, characterized in that, When the slabs are being moved in and out of the warehouse, the trucks stop at the rectangular parking lines marked on the warehouse area; the method for converting each image of the slabs being moved in and out of the warehouse into a bird's-eye view is as follows: Select the coordinates of the rectangular parking lines drawn in the reservoir area in the image, and use the perspective matrix algorithm in OpenCV image processing to convert them into a bird's-eye view.

3. The automatic control method for the entry and exit of vehicle-mounted slabs according to claim 2, characterized in that, The method for creating the dataset required for training the vehicle-mounted slab recognition model is as follows: The image annotation tool labelme was used to segment and annotate the truck and slab sections in the obtained bird's-eye view image. Delete images that are blurry or were captured with significant shaky footage during the annotation process; Convert the JSON file obtained from the labeled image segmentation results into a mask.

4. The automatic control method for the entry and exit of vehicle-mounted slabs according to claim 3, characterized in that, The training method for the image segmentation neural network model is as follows: The segmented and labeled mask image is divided into training and validation sets. By employing a segmentation neural network model, setting hyperparameters, using data augmentation, and after adjustments and optimizations, a segmentation network recognition model was obtained through training.

5. The automatic control method for the entry and exit of vehicle-mounted slabs according to claim 4, characterized in that, The segmentation neural network used is the deeplabv3plus neural network.

6. The automatic control method for the entry and exit of vehicle-mounted slabs according to claim 1, characterized in that, The images of each frame of the truck carrying the slabs are obtained as follows: Cameras were installed in the slab warehouse area to record the daily entry and exit of slabs from vehicles. The video recording of the daily operation of the vehicle-mounted slab is captured and saved locally, and then each frame is extracted from it. Batch process all images, deleting those without trucks or those obscured by vehicles.

7. The automatic control method for the entry and exit of vehicle-mounted slabs according to claim 1, characterized in that, If the centroid coordinates of the image obtained from the segmentation and recognition results only contain the coordinates of the truck, it is considered an outbound task; if the coordinates of the slab are present, it is considered an inbound task.

Images