Small billet tracking system and method with overhead crane and stack position dual perception fusion

Abstract

The present application provides a kind of small billet tracking system and method of crane and stack dual perception fusion, for steelmaking plant small billet yard well-shaped stacking, crane frequency of lifting, billet position tracking precision is insufficient, physical flow and information flow is easy to disengage the industry pain point, fusion crane two-dimensional absolute positioning, clamp state vision-laser fusion perception, three major core technologies of stack surface billet visual identification, realize small billet lifting, hoisting, the position tracking and operation event accurate judgment of whole process of depositing, can accurately position billet in stack row, column, layer information, completely solve the problem of traditional yard dependence artificial record, tracking precision is insufficient, realize the fine, intelligent management of billet yard.

Description

Technical Field

[0001] This invention relates to the field of sensor and tracking technology, specifically to a small billet tracking system and method that integrates the dual sensing of overhead crane and stacking position. Background Technology

[0002] Currently, for tracking small billets, existing technologies employ either a traditional overhead crane positioning system combined with weight sensor detection or a common industrial camera identification method. However, weight sensors cannot determine the clamp rotation angle; and common cameras are prone to ghosting and missed detections in the turbulent environment of frequent overhead crane starts and stops.

[0003] Currently, conventional overhead crane positioning systems can only achieve two-dimensional planar coordinate positioning of the overhead crane's main trolley and auxiliary trolley. Even when combined with mechanical limiters or weight sensors, they still cannot accurately identify the empty / loaded state of the clamps, cannot automatically determine valid lifting events, and cannot identify the placement posture of the billets and the layer, row, and column information of the stack. The tracking accuracy is insufficient, and in scenarios where small square billets are stacked in a grid pattern, billet information mismatch is very likely to occur, making accurate tracking impossible.

[0004] In addition, the crane tracking system with single vision recognition only identifies the clamp status through the vision device at the crane end, without cross-verification based on the actual stacking situation at the stacking site. In scenarios with multiple layers of small billets stacked, the recognition accuracy drops significantly, and it is unable to accurately locate the specific row and column of the billet. The tracking error is large and cannot meet the high reliability requirements of continuous production in steel plants.

[0005] Patent application CN113844813A discloses a slab recognition, tracking, scheduling, and management system based on machine vision, belonging to the field of overhead crane material handling technology. It includes a warehouse management system, a slab scheduling and control system, and a slab vision recognition system. The warehouse management system interfaces with both the slab scheduling and control system and the slab vision recognition system. The warehouse management system adopts a B / S architecture for real-time monitoring and management of slab warehouse equipment and inventory status, as well as scheduling and warehousing planning. The slab scheduling system includes an overhead crane PLC, which uses a 5G base station to establish a communication connection with the overhead crane PLC for controlling overhead crane operation and achieving slab tracking and precise positioning. The slab vision recognition system includes a slab information acquisition subsystem on the roller conveyor, a slab information acquisition subsystem under the overhead crane, and a local image ID recognition and processing subsystem, used to acquire the length, width, and height information of the slabs coming out of the continuous casting machine, slab coordinates, and on-site ID images, respectively. This solution mainly solves the problems of slab position recognition and slab information recognition on the roller conveyor, as well as the position and height recognition of slabs within the warehouse area. The proposed solution suffers from a significant drop in recognition accuracy when multiple layers of small square billets are stacked in a grid pattern. This patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a small billet tracking system and method that integrates the dual sensing capabilities of the overhead crane and the stacking position.

[0007] The billet tracking system based on the dual-sensor fusion of overhead crane and stacking position provided by the present invention includes: an overhead crane position detection device, an overhead crane sensing system, a stacking position sensing system, a network transmission unit, and a billet warehouse management system. The overhead crane position detection device includes a trolley X-axis position detection module and a trolley Y-axis position detection module. The trolley X-axis position detection module and the trolley Y-axis position detection module are spatially perpendicularly distributed and used together to form a unique position identifier of the overhead crane in the warehouse area plane. The overhead crane perception system includes a laser scanner, a camera, and an on-board controller. The laser scanner, camera, and on-board controller are all integrated and installed on the overhead crane. The laser scanner is used to collect three-dimensional point cloud data of the clamp and the billet below in real time. The camera is used to collect video footage of the clamp operation in real time. The on-board controller is used to receive the three-dimensional point cloud data and the video footage. The stacking position sensing system includes multiple fixedly installed cameras and an image processing unit. The cameras are installed next to or above the stacking positions in the warehouse area, with the lenses aimed at the target stacking position, to collect image data of the surface blanks of the stacking position in real time and transmit it to the image processing unit. The network transmission unit is used to realize data and video transmission between the crane position detection device, the crane sensing system, the stacking position sensing system and the billet warehouse management system. The billet warehouse management system is connected to the overhead crane position detection device, the overhead crane sensing system, the stacking position sensing system, and the network transmission unit, respectively, to receive and process positioning data, sensing data, and identification data, and to update the warehouse location and billet information.

[0008] Preferably, the on-board controller in the crane perception system is configured to: synchronously input the three-dimensional point cloud data and the video footage, identify the empty / loaded state and rotation state of the clamps through a visual model and point cloud analysis algorithm, and determine the lifting point, lifting time, unloading point and unloading time of the hoisting event by combining the crane positioning data; The visual model adopts an instance segmentation model, expressed as follows:

[0009] in, The output of the visual model, The input video frame image at time t, Pre-train weights for the model; For instance segmentation models, denoted by , represents the mapping function for instance segmentation inference on the input image; The model outputs the bounding box of the clamp jaws. , These are the x and y coordinates of the top-left corner of the bounding box in the image, respectively. These are the width and height of the bounding box, respectively; Using a deep learning model, the model performs frame-by-frame inference on video frames captured in real-time by the camera. The model focuses on the opening angle and closing state of the clamp jaws. Under no-load conditions, the clamps are fully open. Under heavy load conditions, the clamps exhibit a specific micro-opening angle due to holding the billet, and the jaws and the billet surface form a clear contact edge in the image. The model learns the correspondence between jaw morphology and load in labeled samples and outputs the load classification confidence score of the clamps. 0 = no load, 1 = heavy load; The point cloud analysis algorithm includes: Define the clamp point cloud set as The point cloud set in the lower region is ; The load state is determined jointly by spatial overlap and normal vector consistency:

[0010] in, The result is the load condition determination result; This is an overlap calculation function used to calculate the overlap ratio between the jaw point cloud and the point cloud below in the Z-axis projection. This is a function for calculating the similarity of normal vectors, used to calculate the cosine similarity of the angle between the normal vector of the clamp point cloud and the normal vector of the point cloud below. The preset spatial overlap threshold; The normal vector of the clamp point cloud; Let be the normal vector of the point cloud below; This is the preset threshold for normal vector similarity.

[0011] Preferably, the vehicle controller is further configured to fuse the recognition results from visual and point cloud methods using a weighted voting method:

[0012] in, , These are the weighting coefficients; if If the load exceeds a preset threshold, it is considered an overload. Determine a hoisting event using the following steps: The clamp status is monitored in real time. When a change from no-load to heavy-load is detected, it is marked as a potential lifting event. When a change from heavy-load to no-load is detected, it is marked as a potential unloading event. Based on the XY axis position data of the overhead crane, determine whether the location of the status change is within the effective working area; The crane's trajectory before and after the status change was confirmed to conform to the hoisting logic. After confirmation, record the event type, time of occurrence, and location of occurrence, and send it to the billet warehouse management system.

[0013] Preferably, the image processing unit in the stacking position sensing system is configured to: identify the number of rows and columns of the topmost square billet in the stacking position using an instance segmentation model; The instance segmentation model outputs the bounding box, confidence score, and instance mask for each detected billet instance, and obtains the number of rows or columns of the upper-layer billet by counting the detected instances.

[0014] Preferably, the image processing unit in the stacking position sensing system is further configured as follows: Initially, a baseline is established for empty stack positions or stack positions with known layouts through an instance segmentation model, and the instance mask, center point coordinates and corresponding row and column indices of each billet in the upper layer are recorded; After each hoisting event is completed, the set of billet instances at the top of the current stack and their row and column layout are automatically saved. The current stack position information is segmented using a model and compared with historical states. If the maximum overlap between the mask of a billet instance in the historical image and the masks of all billet instances in the current image is less than a preset threshold, it is determined that the billet has been lifted away and is marked as a changed instance. Combining the stack position's historical rows and columns, the current row / column information of the changed instance is obtained and the current information is updated in the stack position status.

[0015] Preferably, in the crane position detection device: When the X-axis position detection module uses continuous position detection sensors, including Gray busbars, absolute value encoders, and barcode strips, the sensors are fixed on ground supports on the side of the trolley track and laid along the entire length of the trolley track. The address encoder is installed on the end beam of the trolley. When the X-axis position detection module uses non-contact position detection sensors, including lasers and microwave radars, the sensor transmitter is installed on the trolley, and the reflector is fixed at the end of the track, or the installation positions of the sensor transmitter and the reflector are interchanged. When the Y-axis position detection module uses continuous position detection sensors, including Gray busbars, absolute value encoders, and barcode strips, the sensors are fixed on the support of the main beam of the overhead crane and laid along the entire length of the overhead crane trolley running track, with the address encoder installed on the overhead crane trolley. When the Y-axis position detection module uses non-contact position detection sensors, including lasers and microwave radars, the sensor transmitter is installed at the end of the trolley, and the reflector is fixed at the end of the main beam, or the installation positions of the sensor transmitter and the reflector are interchanged. The control system receives the detection values ​​from the position detection sensor in real time and converts the detection values ​​into X / Y axis coordinates in the reservoir area coordinate system.

[0016] Preferably, the vehicle controller is further configured to perform Euclidean distance clustering on the point cloud: The point cloud set after filtering and downsampling The clustering conditions are:

[0017] in, For point cloud collection The nth point cloud; This represents the i-th point cloud cluster obtained after clustering; Let be any point in the point cloud that is currently being determined to belong to this cluster. It already belongs to this cluster For any point in the cloud, j and k are the indexes of the point cloud sequence; This is the clustering distance threshold; Principal component analysis was performed on the segmented clamp point cloud to calculate its covariance matrix C:

[0018] Perform eigenvalue decomposition on C:

[0019] Eigenvalues Corresponding feature vector These represent the main directions of the point cloud, namely the length, width, and height axes; in, The total number of points in the clamp point cloud; Let i be the i-th point in the clamp point cloud; is the mean vector of the point cloud; It is the transpose symbol; Let be the j-th eigenvector of the covariance matrix C, corresponding to the principal direction of the point cloud; Let J be the j-th eigenvalue of the covariance matrix C; Calculate the minimum bounding box by taking the length corresponding to the direction with the largest span, the width corresponding to the direction with the second largest span, and the height corresponding to the direction with the smallest span.

[0020] Preferably, the billet warehouse management system is configured to: combine the clamp rotation status and lifting operation events identified by the crane sensing system, and the changes in the stack position image before and after the crane lifting identified by the stack position sensing system, to determine the row or column information of the billet in the stack position for this crane operation, so as to achieve accurate anchoring of the billet position, and update the warehouse position information according to the lifting operation of the crane.

[0021] The method for tracking small billets by dual sensing fusion of overhead crane and stacking position provided by the present invention includes the following steps: Step 1: System initialization. The billet warehouse management system completes the mapping configuration between logical and physical warehouse locations, configures the basic information of billets, and initializes system parameters. The crane position detection device, crane sensing system, and stack position sensing system complete self-checks, collect and upload basic data in real time. Step 2: The crane positioning and detection device collects the absolute position data of the crane's X-axis and Y-axis in real time, and the crane's perception system collects the 3D point cloud and video images of the clamps in real time. The model of the on-board controller identifies the clamps' no-load / heavy-load status and rotation angle in real time. Step 3: The on-board controller combines the crane position data and clamp status data. When it detects that the clamp has changed from unloaded to heavily loaded and the position data of the trolley and crane match the lifting action logic, it sends the lifting event recognition result to the billet warehouse management system. Step 4: The billet warehouse management system receives the lifting event identification result and locks the warehouse location information corresponding to the lifting position; Step 5: After the crane leaves the work position, the stack position sensing system collects images of the stack position, identifies the number of rows and columns of the top layer of blanks in the stack position through a visual segmentation model, and compares them with photos after the previous hoisting operation to identify changes in the stack position images. Step 6: The billet warehouse management system combines the rotation status of the overhead crane clamps to complete the multi-source fusion of overhead crane positioning data, clamp status data, and stack position identification data, determines the target lifting position and layer, row, and column information of this operation, and simultaneously retrieves the billet basic information corresponding to that position to complete the binding of the lifting event with the billet object; Step 7: Repeat step 2; Step 8: The on-board controller combines the crane position data and clamp status data. When it detects that the clamp has changed from heavy load to no load and the position data of the trolley and the crane match the unloading action logic, it sends the unloading event recognition result to the billet warehouse management system. Step 9: Repeat step 5; Step 10: The billet warehouse management system combines the rotation status of the overhead crane clamps to complete the multi-source fusion of overhead crane positioning data, clamp status data, and stack position identification data, determines the target unloading position and layer, row, and column information of this operation, and simultaneously retrieves the corresponding billet basic information, automatically updating the warehouse occupancy information and billet position information in the system. Step 11: Repeat steps 2 to 10 to automatically track all overhead crane operations in the warehouse area, including inbound, unpacking, transfer, and outbound operations, and simultaneously feed the actual operation data back to the upper-level information system.

[0022] Preferably, in step 5, the stack position sensing system identifies the number of rows and columns of the topmost blank in the stack position using an instance segmentation model, specifically including: A spatial index is constructed for the preprocessed point cloud, and distance thresholds, minimum cluster point count, and maximum cluster point count are set. By clustering, points with a spatial distance less than a threshold are grouped into the same cluster, and multiple point cloud clusters are output. Each cluster is traversed, the bounding box is calculated, and the cluster corresponding to the largest bounding box within the vehicle's position range is used as the clamp point cloud data. In load condition determination, if the normal vector of the point cloud in the jaw region is parallel to the normal vector of the adjacent point cloud below and the distance is within a preset threshold, it is determined to be overloaded.

[0023] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention solves the problem that the traditional overhead crane positioning system can only obtain the two-dimensional coordinates of the large and small trolleys, and cannot accurately identify the row and column information of the billet in the grid-shaped stack, and cannot adapt to the tracking needs of the multi-layer stacking operation of small square billets in steel plants; (2) This invention solves the industry pain points of billet placement posture being unrecognizable due to the rotation of the overhead crane clamp, incorrect judgment of hoisting events, and mismatch of billet tracking information; (3) The present invention solves the problems of traditional warehouse manual inventory, reliance on manual recording of hoisting information, high labor intensity and extremely low efficiency of blank finding operation, disconnect between physical flow and information flow, and inability to achieve refined warehouse management; (4) This invention solves the problem that the mechanical structure of the overhead crane shakes under high temperature and high dust environment in steelmaking plants, resulting in low recognition accuracy of traditional mechanical limit or single vision scheme; through multi-dimensional perception, it eliminates misjudgment caused by clamp rotation, swing, and occlusion, and ensures a high degree of consistency between physical flow and information flow. Attached Figure Description

[0024] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 Here is a structural diagram of the billet warehouse management system; Figure 2 A diagram showing the installation locations of the camera and laser scanner; Figure 3 Diagram showing the camera installation locations on the stack; Figure 4 A complete flowchart of the system's hardware and software collaboration; The diagram shows: 1-Crane position detection device, 2-Crane sensing system, 3-Stacking position sensing system, 4-Network transmission unit, 5-Bill warehouse management system, 6-Camera on the crane, 7-LiDAR on the crane, 8-Camera next to the stacking position. Detailed Implementation

[0025] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0026] Example This invention provides a small billet tracking system integrating overhead crane and stacking position sensing, comprising five core units: an overhead crane position detection device 1, an overhead crane sensing system 2, a stacking position sensing system 3, a network transmission unit 4, and a billet warehouse management system 5 (e.g., ...). Figure 1 As shown in the figure, the structure, connection relationship and working principle of each unit are as follows: Overhead crane position detection device 1: It is divided into a main crane X-axis position detection module and a trolley Y-axis position detection module; the X-axis and Y-axis position detection modules can be combined with the warehouse area conditions and use Gray busbars, lasers and other ranging devices to realize the absolute position detection of the overhead crane and trolley in the direction of the main crane and trolley.

[0027] Positional relationship between X-axis and Y-axis position detection modules: The crane position detection system adopts an orthogonal two-dimensional absolute coordinate system. The X-axis and Y-axis modules are spatially perpendicular and together constitute a unique position identifier for the crane on the warehouse area plane. X-axis (cabin direction): If the Gray busbar scheme is adopted, the Gray busbar body is fixed on the ground support on the side of the trolley track and laid along the entire length of the trolley track, corresponding to the horizontal movement position of the trolley along the track. The address encoder is installed on the end beam of the trolley; if the laser ranging scheme is adopted, the laser emitter is installed on the trolley, and the reflector is fixed on the column at the end of the track.

[0028] Y-axis (trolley direction): If the Gray busbar scheme is adopted, the Gray busbar body is fixed on the support of the overhead crane main beam and laid along the entire length of the overhead crane trolley running track, corresponding to the horizontal movement position of the trolley along the track. The address encoder is installed on the overhead crane trolley. In the laser ranging scheme, the laser emitter is installed at the end of the trolley, and the reflector is fixed at the end of the main beam; the reverse is also possible.

[0029] Spatial coordinate origin: For example, set the crane track at a certain corner of the reservoir area as the starting point, with the positive direction of the X-axis pointing east, the positive direction of the Y-axis pointing south, and the positive direction of the Z-axis (height direction) pointing upward, completely aligned with the geographical coordinate system of the reservoir area.

[0030] The specific process of XY axis position detection: The control system receives the detection values ​​from position detection sensors (such as Gray line, laser rangefinder, etc.) in real time, and converts the detection values ​​into X / Y axis coordinates in the reservoir area coordinate system to correspond to the geographical coordinates of the reservoir area.

[0031] Among the methods used for crane position detection are absolute value encoding rulers, RFID, rotary encoders, microwave radar, etc.

[0032] Crane Sensing System 2: Includes a laser scanner, camera, and onboard controller; all devices are integrated and installed on the crane (the installation positions of camera 6 and laser scanner 7 are shown in the image). Figure 2 (As shown). The laser scanner acquires real-time 3D point cloud data of the clamp and the billet below, while the camera acquires real-time video footage of the clamp operation. Both types of data are simultaneously input into the onboard controller. Through visual models and point cloud analysis algorithms, the controller identifies the clamp's no-load / load status and rotation status in real time. Combined with the crane positioning data, it determines the lifting point, lifting time, unloading point, and unloading time of the hoisting event, providing crane action event trigger signals for billet tracking.

[0033] In some preferred embodiments, point cloud data and video footage can be transmitted to the ground first, and then received by a ground server.

[0034] The main principles are as follows: The visual model includes clamp recognition and state determination based on the visual model: Using the YOLOv8 instance segmentation model, the expression is:

[0035] in, The output of the visual model, The input video frame image at time t, Pre-train weights for the model; For the YOLOv8 instance segmentation model, denoted as the mapping function for instance segmentation inference on the input image; The model outputs the bounding box of the clamp jaws. , These are the x and y coordinates of the top-left corner of the bounding box in the image, respectively. These are the width and height of the bounding box, respectively; No-load / Heavy-load identification: Using the YOLOv8 deep learning model, frame-by-frame inference is performed on video frames captured in real time by the camera. The model focuses on the opening angle and closing state of the clamp jaws. Under no-load conditions, the clamps are usually fully open; under heavy load conditions, the clamps exhibit a specific opening angle due to holding the billet, and the jaws and the billet surface form a clear contact edge in the image. The visual model learns the correspondence between jaw shape and load in a large number of labeled samples and outputs the load classification confidence score of the clamps.

[0036] Load classification confidence 0 = no load, 1 = heavy load.

[0037] The point cloud analysis algorithm includes clamp 3D pose and load sensing based on point cloud analysis: Point cloud preprocessing and segmentation: After filtering (pass-through filtering, outlier removal) and downsampling, point cloud clusters belonging to the clamp and the region below it are segmented from the background point cloud using Euclidean distance-based clustering. Define the clamp point cloud set as The point cloud set in the lower region is ; The load state is determined jointly by spatial overlap and normal vector consistency:

[0038] in, The result is the load condition determination result; The method for calculating overlap is as follows: taking a set of point clouds as a reference, it checks whether each point in the set of point clouds has a "nearest neighbor" point in another set of point clouds with a distance less than a preset threshold; if such a neighbor exists, the two points are considered to overlap, and then the overall overlap rate of the two sets of point clouds is calculated. Formula for calculating the normal vector consistency measure: ; in, Represents the vector dot product. Represents the magnitude of a vector; when The closer the value is to 1, the better the clamp is in contact with the surface of the object below at that point; the closer the value is to 0, the more perpendicular the two planes are. The preset spatial overlap threshold; The normal vector of the clamp point cloud; Let be the normal vector of the point cloud below; This is the preset threshold for normal vector similarity.

[0039] Clamp status recognition process: Data synchronous acquisition: The laser scanner and camera achieve precise timestamp alignment through trigger signals, and simultaneously acquire 3D point clouds and video frames of the clamp area.

[0040] Visual model inference: Input video frames into the YOLOv8 model and output the bounding box of the clamp and the initial load judgment (no load / heavy load). and confidence level.

[0041] Point cloud data processing: The acquired point cloud is filtered, downsampled, and segmented to extract point cloud clusters in the clamp and the area below it, and calculations are performed. ; Fusion Decision: A weighted voting method is used to fuse the recognition results from visual and point cloud data.

[0042] in, , The weighting coefficients are (∈(0,1), determined through experimental optimization); if If the load exceeds the preset threshold, it is considered an overload.

[0043] Lifting event judgment process: Status change detection: Real-time monitoring of clamp status. When a status change from no-load to heavy-load is detected, it is marked as a potential lifting event; when a status change from heavy-load to no-load is detected, it is marked as a potential unloading event.

[0044] Position logic verification: Combine the XY axis position data of the overhead crane to determine whether the location of the status change is within the effective working area (such as above the stack or above the roller conveyor).

[0045] Action sequence verification: Confirm that the movement trajectory of the overhead crane before and after the state change conforms to the hoisting logic (e.g., the overhead crane is stationary before hoisting, and starts to move after hoisting).

[0046] Event Confirmation and Output: If all the above conditions are met, the hoisting event is confirmed to have occurred. The event type, occurrence time, and occurrence location are recorded and sent to the billet warehouse management system.

[0047] Euclidean distance clustering: for the filtered and downsampled set of point clouds The clustering conditions are:

[0048] in, For point cloud collection The nth point cloud; This represents the i-th point cloud cluster obtained after clustering; Let be any point in the point cloud that is currently being determined to belong to this cluster. It already belongs to this cluster For any point in the cloud, j and k are the indexes of the point cloud sequence; This is the clustering distance threshold; The specific process of segmentation: Preprocessing: Apply pass-through filtering and radius filtering to remove outliers, downsample, and reduce the amount of data.

[0049] Segmentation: Background point cloud is identified and removed using a clustering algorithm based on Euclidean distance.

[0050] A spatial index (KD tree) is constructed on the preprocessed point cloud.

[0051] Set the distance threshold dist, the minimum number of cluster points mincount, and the maximum number of cluster points maxcount.

[0052] Clustering is used to group points with a spatial distance less than a threshold into the same cluster, outputting multiple point cloud clusters. Each cluster is traversed, and the bounding box is calculated. The largest bounding box within the vehicle's position range is considered to be the cluster where the clamp belongs, thus obtaining the clamp's point cloud data.

[0053] Geometric feature extraction: Principal component analysis (PCA) is performed on the segmented clamp point cloud to calculate the length, width, height, and spatial orientation of its minimum bounding box. Under heavy load, billet point cloud (appearing as regular rectangular or strip-shaped point cloud blocks) will continuously appear below the clamp point cloud, and the clamp jaw point cloud and the billet surface point cloud are spatially closely attached; under no load, the point cloud below the clamp is sparse or only ground / other interfering point clouds exist.

[0054] Principal Component Analysis (PCA) Expression: Pliers point cloud Calculate its covariance matrix C:

[0055] Perform eigenvalue decomposition on C:

[0056] Eigenvalues Corresponding feature vector This refers to the main directions of the point cloud (length, width, and height axes).

[0057] in, Let be the coordinates of the i-th point in the clamp point cloud; The total number of points in the clamp point cloud; Let i be the i-th point in the clamp point cloud; is the mean vector of the point cloud; It is the transpose symbol; Let be the j-th eigenvector of the covariance matrix C, corresponding to the principal direction of the point cloud; Let J be the j-th eigenvalue of the covariance matrix C; Minimum bounding box calculation process: The centered point cloud is projected onto the three feature vector directions, and the projection length in each direction is calculated.

[0058] Regarding direction Projection value:

[0059] in, For the i-th centralized point cloud in the direction Projection value on; This represents the i-th point cloud after centralization. Find the minimum value respectively and maximum value Then the span in that direction for:

[0060] Length: Take the length corresponding to the direction with the largest span, i.e., L=d1 (corresponding to v1). Width: Take the length corresponding to the direction with the second largest span, i.e., W=d2 (corresponding to v2); Height: Take the length corresponding to the direction with the smallest span, i.e., H=d3 (corresponding to v3); Load status determination: By analyzing the spatial topological relationship between the clamp point cloud and the point cloud below: if the normal vector of the point cloud in the clamp area is parallel to the normal vector of the adjacent point cloud below and the distance is within a preset threshold (indicating contact), it is determined to be overloaded; otherwise, it is unloaded. If the clamp is overloaded, its jaws will directly contact the billet, theoretically the point cloud distance is 0. Considering the laser scanning resolution and data return effect, this distance needs to be increased, currently tentatively set at 100mm, and will be adjusted based on actual conditions.

[0061] Rotation direction determination: The initial rotation state of the clamp is defined by the direction of the clamp length. The direction v1 detected by the PCA is the current rotation direction of the clamp.

[0062] Stacking position sensing system 3: consists of multiple fixedly installed cameras (the installation positions of the cameras 8 on the stacking position are as follows) Figure 3 The system (as shown) consists of an image processing unit and a camera installed at a preset height and angle next to or above the stack in the warehouse area. The lens is aimed at the target stack, and the lens's field of view should be able to cover the target stack. The camera acquires image data of the surface billets of the stack in real time and transmits it to the image processing unit. Through an instance segmentation model, the number of rows and columns of the topmost square billets in the stack is identified.

[0063] The image is processed by a trained model to obtain the bounding box, confidence score, and instance mask (binary image, the same size as or scaled from the original image) for each detected billet instance.

[0064] Filtering the topmost billet: Since only the topmost billet was labeled during training, the model only outputs the topmost instance. If the scene is partially occluded, but the topmost billet is fully visible, the model can still segment the visible part. By counting the detected instances, the number of rows or columns of the topmost billet can be obtained.

[0065] The main principles are as follows: a) Historical state modeling Initially, a baseline is established for empty stack positions or stack positions with known layouts using an instance segmentation model: the instance mask, center point coordinates, and corresponding row and column indices of each billet in the upper layer are recorded.

[0066] After each hoisting event is completed, the system automatically saves the set of billet instances at the top of the current stack and their row and column layout.

[0067] b) Before-and-after state comparison process Change detection: The current stack position information is segmented using a model and compared with historical states. If a billet instance in a historical image has no corresponding mask in the current image (it has been removed), or a new mask appears (a new billet is placed), it is marked as a change instance. Combining the historical rows and columns of the stack position, the current row / column information of the change instance is obtained, and the current information is updated in the stack position status.

[0068] In some preferred embodiments, the functional responsibilities of the crane sensing system 2 and the stacking position sensing system 3 are assignable, with the crane sensing system 2 performing some of the functions of the stacking position sensing system 3, or the stacking position sensing system 3 performing some of the functions of the crane sensing system 2.

[0069] Network transmission unit 4: Enables data and video transmission between the crane position detection device, crane sensing system, stack position sensing system, network transmission unit, and billet warehouse management system.

[0070] The billet warehouse management system 5 is the core control and data processing hub of the entire system. The system receives and processes positioning, sensing, and identification data uploaded by each unit, completes data fusion and logical judgment, and updates warehouse location and billet information. Its main principle is to combine the clamp rotation status and lifting operation events identified by the overhead crane sensing system with the changes in the stack position image before and after the overhead crane lifting, accurately determining the row or column information of the billet in the stack position for this overhead crane operation. This achieves precise anchoring of the billet position and updates the warehouse location information based on the lifting operation of the overhead crane.

[0071] like Figure 4 The complete workflow of hardware and software collaboration in this system is as follows: Step 1: System initialization. The billet warehouse management system completes the mapping configuration between logical and physical warehouse locations based on the warehouse layout, and configures the basic information of billets (if there is a communication interface with the upper-level information system, it can also receive the basic information of billets from the upper-level information system), and completes the initialization of system parameters; the crane position detection device, crane sensing system, and stack position sensing system complete self-checks, and collect and upload basic data in real time.

[0072] Step 2: Real-time positioning and status monitoring of the overhead crane. The overhead crane positioning and detection device collects the absolute position data of the X-axis of the main trolley and the Y-axis of the trolley in real time. The overhead crane perception system collects the 3D point cloud and video images of the clamps in real time. The model of the on-board controller identifies the no-load / heavy-load status and rotation angle of the clamps in real time.

[0073] Step 3: Automatic identification and triggering of lifting events. The on-board controller combines the crane position data and clamp status data. When it is detected that the clamp has changed from empty to heavy load and the position data of the trolley and the crane match the lifting action logic, the lifting event identification result is sent to the billet warehouse management system.

[0074] Step 4: The billet warehouse management system receives the lifting event identification result and locks the warehouse location information (such as warehouse location, roller conveyor and other ground equipment) corresponding to the lifting position.

[0075] Step 5: After the overhead crane leaves the work position, the stack position sensing system acquires images of the stack position and identifies the number of rows and columns of the top layer of blanks in the stack position using a visual segmentation model. This is then compared with photos taken after the previous hoisting operation when the overhead crane left to identify changes in the stack position image.

[0076] Step 6: The billet warehouse management system, by combining the rotation status of the overhead crane clamps, completes the multi-source fusion of overhead crane positioning data, clamp status data, and stack position identification data to determine the target lifting position and layer, row, and column information for this operation. Simultaneously, it retrieves the corresponding billet foundation information for that position, completing the binding of the lifting event with the billet object.

[0077] Step 7: Repeat step 2.

[0078] Step 8: Automatic identification and triggering of unloading events. The vehicle controller combines the crane position data and clamp status data. When it is detected that the clamp has changed from heavy load to no load and the position data of the crane and trolley match the unloading action logic, the unloading event identification result is sent to the billet warehouse management system.

[0079] Step 9: Repeat step 5.

[0080] Step 10: The billet warehouse management system, by combining the rotation status of the overhead crane clamps, completes multi-source fusion of overhead crane positioning data, clamp status data, and stack location identification data to determine the target unloading location and layer, row, and column information for this operation. Simultaneously, it retrieves the corresponding billet basic information and automatically updates the warehouse occupancy information and billet location information within the system, completing the automatic tracking of this hoisting operation.

[0081] Step 11: Closed-loop management of the entire process. The system repeats steps 2 to 10 to automatically track all types of operations of the overhead cranes in the warehouse area, such as warehousing, stacking, transfer, and warehousing. The system also feeds back the actual operation data to the upper-level information system to complete data archiving, operation history recording, operation report statistics, and abnormal fault alarms.

[0082] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0083] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A small billet tracking system integrating overhead crane and stacking position sensing, characterized in that, include: Crane position detection device, crane sensing system, stacking position sensing system, network transmission unit and billet warehouse management system; The overhead crane position detection device includes a trolley X-axis position detection module and a trolley Y-axis position detection module. The trolley X-axis position detection module and the trolley Y-axis position detection module are spatially perpendicularly distributed and used together to form a unique position identifier of the overhead crane in the warehouse area plane. The overhead crane perception system includes a laser scanner, a camera, and an on-board controller. The laser scanner, camera, and on-board controller are all integrated and installed on the overhead crane. The laser scanner is used to collect three-dimensional point cloud data of the clamp and the billet below in real time. The camera is used to collect video footage of the clamp operation in real time. The on-board controller is used to receive the three-dimensional point cloud data and the video footage. The stacking position sensing system includes multiple fixedly installed cameras and an image processing unit. The cameras are installed next to or above the stacking positions in the warehouse area, with the lenses aimed at the target stacking position, to collect image data of the surface blanks of the stacking position in real time and transmit it to the image processing unit. The network transmission unit is used to realize data and video transmission between the crane position detection device, the crane sensing system, the stack position sensing system and the billet warehouse management system. The billet warehouse management system is connected to the overhead crane position detection device, the overhead crane sensing system, the stacking position sensing system, and the network transmission unit, respectively, to receive and process positioning data, sensing data, and identification data, and to update the warehouse location and billet information.

2. The small billet tracking system based on the dual-sensor fusion of overhead crane and stacking position as described in claim 1, characterized in that, The onboard controller in the overhead crane perception system is configured to: synchronously input the three-dimensional point cloud data and the video footage; identify the empty / loaded state and rotation state of the clamps through a visual model and point cloud analysis algorithm; and determine the lifting point, lifting time, unloading point, and unloading time of the hoisting event by combining the overhead crane positioning data. The visual model adopts an instance segmentation model, expressed as follows: in, The output of the visual model, The input video frame image at time t, Pre-train weights for the model; For instance segmentation models, denoted by , represents the mapping function for instance segmentation inference on the input image; The model outputs the bounding box of the clamp jaws. , These are the x and y coordinates of the top-left corner of the bounding box in the image, respectively. These are the width and height of the bounding box, respectively; Using a deep learning model, the model performs frame-by-frame inference on video frames captured in real-time by the camera. The model focuses on the opening angle and closing state of the clamp jaws. Under no-load conditions, the clamps are fully open. Under heavy load conditions, the clamps exhibit a specific opening angle due to holding the billet, and the jaws and the billet surface form a clear contact edge in the image. The model learns the correspondence between jaw shape and load in labeled samples and outputs the load classification confidence score of the clamps. 0 = no load, 1 = heavy load; The point cloud analysis algorithm includes: Define the clamp point cloud set as The point cloud set in the lower region is ; The load state is determined jointly by spatial overlap and normal vector consistency: in, The result is the load condition determination result; This is an overlap calculation function used to calculate the overlap ratio between the jaw point cloud and the point cloud below in the Z-axis projection. This is a function for calculating the similarity of normal vectors, used to calculate the cosine similarity of the angle between the normal vector of the clamp point cloud and the normal vector of the point cloud below. The preset spatial overlap threshold; The normal vector of the clamp point cloud; Let be the normal vector of the point cloud below; This is the preset threshold for normal vector similarity.

3. The small billet tracking system integrating overhead crane and stacking position dual sensing according to claim 2, characterized in that, The vehicle controller is also configured to use a weighted voting method to fuse the recognition results from visual and point cloud data. in, , These are the weighting coefficients; if If the load exceeds a preset threshold, it is considered an overload. Determine a hoisting event using the following steps: The clamp status is monitored in real time. When a change from no-load to heavy-load is detected, it is marked as a potential lifting event. When a change from heavy-load to no-load is detected, it is marked as a potential unloading event. Based on the XY axis position data of the overhead crane, determine whether the location of the status change is within the effective working area; The crane's trajectory before and after the status change was confirmed to conform to the hoisting logic. After confirmation, record the event type, time of occurrence, and location of occurrence, and send it to the billet warehouse management system.

4. The small billet tracking system with dual sensing fusion of overhead crane and stacking position as described in claim 1, characterized in that, The image processing unit in the stacking position sensing system is configured to identify the number of rows and columns of the topmost square billet in the stacking position using an instance segmentation model. The instance segmentation model outputs the bounding box, confidence score, and instance mask for each detected billet instance, and obtains the number of rows or columns of the upper-layer billet by counting the detected instances.

5. The small billet tracking system integrating overhead crane and stacking position dual sensing according to claim 4, characterized in that, The image processing unit in the stacking site sensing system is further configured to: Initially, a baseline is established for empty stack positions or stack positions with known layouts through an instance segmentation model, and the instance mask, center point coordinates and corresponding row and column indices of each billet in the upper layer are recorded; After each hoisting event is completed, the set of billet instances at the top of the current stack and their row and column layout are automatically saved. The current stack position information is segmented using a model and compared with historical states. If the maximum overlap between the mask of a billet instance in the historical image and the masks of all billet instances in the current image is less than a preset threshold, it is determined that the billet has been lifted away and is marked as a changed instance. Combining the stack position's historical rows and columns, the current row / column information of the changed instance is obtained and the current information is updated to the stack position status.

6. The small billet tracking system integrating overhead crane and stacking position dual sensing according to claim 1, characterized in that, In the crane position detection device: When the X-axis position detection module uses continuous position detection sensors, including Gray busbars, absolute value encoders, and barcode strips, the sensors are fixed on ground supports on the side of the trolley track and laid along the entire length of the trolley track. The address encoder is installed on the end beam of the trolley. When the X-axis position detection module uses non-contact position detection sensors, including lasers and microwave radars, the sensor transmitter is installed on the trolley, and the reflector is fixed at the end of the track, or the installation positions of the sensor transmitter and the reflector are interchanged. When the Y-axis position detection module uses continuous position detection sensors, including Gray busbars, absolute value encoders, and barcode strips, the sensors are fixed on the support of the main beam of the overhead crane and laid along the entire length of the overhead crane trolley running track, with the address encoder installed on the overhead crane trolley. When the Y-axis position detection module uses non-contact position detection sensors, including lasers and microwave radars, the sensor transmitter is installed at the end of the trolley, and the reflector is fixed at the end of the main beam, or the installation positions of the sensor transmitter and the reflector are interchanged. The control system receives the detection values ​​from the position detection sensor in real time and converts the detection values ​​into X / Y axis coordinates in the reservoir area coordinate system.

7. The small billet tracking system based on the dual-sensor fusion of overhead crane and stacking position as described in claim 2, characterized in that, The vehicle controller is also configured to perform Euclidean distance clustering on the point cloud: The point cloud set after filtering and downsampling The clustering conditions are: in, For point cloud collection The nth point cloud in; This represents the i-th point cloud cluster obtained after clustering; Let be any point in the point cloud that is currently being determined to belong to this cluster. It already belongs to this cluster For any point in the cloud, j and k are the indexes of the point cloud sequence; This is the clustering distance threshold; Principal component analysis was performed on the segmented clamp point cloud to calculate its covariance matrix C: Perform eigenvalue decomposition on C: Eigenvalues corresponding feature vector These represent the main directions of the point cloud, namely the length, width, and height axes; in, The total number of points in the clamp point cloud; Let i be the i-th point in the clamp point cloud; is the mean vector of the point cloud; It is the transpose symbol; Let be the j-th eigenvector of the covariance matrix C, corresponding to the principal direction of the point cloud; Let J be the j-th eigenvalue of the covariance matrix C; Calculate the minimum bounding box by taking the length corresponding to the direction with the largest span, the width corresponding to the direction with the second largest span, and the height corresponding to the direction with the smallest span.

8. The small billet tracking system with dual sensing fusion of overhead crane and stacking position as described in claim 1, characterized in that, The billet warehouse management system is configured to: combine the clamp rotation status and lifting operation events identified by the overhead crane sensing system, and the changes in the stack position image before and after the overhead crane lifting identified by the stack position sensing system, to determine the row or column information of the billet in the stack position for this overhead crane operation, thereby achieving accurate anchoring of the billet position, and updating the warehouse position information according to the lifting operation of the overhead crane.

9. A method for tracking small billets based on a crane and stacking position dual-sensor fusion tracking system according to any one of claims 1 to 8, characterized in that, Includes the following steps: Step 1: System initialization. The billet warehouse management system completes the mapping configuration between logical and physical warehouse locations, configures the basic information of billets, and initializes system parameters. The crane position detection device, crane sensing system, and stack position sensing system complete self-checks, collect and upload basic data in real time. Step 2: The crane positioning and detection device collects the absolute position data of the crane's X-axis and Y-axis in real time, and the crane's perception system collects the 3D point cloud and video images of the clamps in real time. The model of the on-board controller identifies the clamps' no-load / heavy-load status and rotation angle in real time. Step 3: The on-board controller combines the crane position data and clamp status data. When it detects that the clamp has changed from unloaded to heavily loaded and the position data of the trolley and crane match the lifting action logic, it sends the lifting event recognition result to the billet warehouse management system. Step 4: The billet warehouse management system receives the lifting event identification result and locks the warehouse location information corresponding to the lifting position; Step 5: After the crane leaves the work position, the stack position sensing system collects images of the stack position, identifies the number of rows and columns of the top layer of blanks in the stack position through a visual segmentation model, and compares them with photos after the previous hoisting operation to identify changes in the stack position images. Step 6: The billet warehouse management system combines the rotation status of the overhead crane clamps to complete the multi-source fusion of overhead crane positioning data, clamp status data, and stack position identification data, determines the target lifting position and layer, row, and column information of this operation, and simultaneously retrieves the billet basic information corresponding to that position to complete the binding of the lifting event with the billet object; Step 7: Repeat step 2; Step 8: The on-board controller combines the crane position data and clamp status data. When it detects that the clamp has changed from heavy load to no load and the position data of the trolley and the crane match the unloading action logic, it sends the unloading event recognition result to the billet warehouse management system. Step 9: Repeat step 5; Step 10: The billet warehouse management system combines the rotation status of the overhead crane clamps to complete the multi-source fusion of overhead crane positioning data, clamp status data, and stack position identification data, determines the target unloading position and layer, row, and column information of this operation, and simultaneously retrieves the corresponding billet basic information, automatically updating the warehouse occupancy information and billet position information in the system. Step 11: Repeat steps 2 to 10 to automatically track all overhead crane operations in the warehouse area, including inbound, unpacking, transfer, and outbound operations, and simultaneously feed the actual operation data back to the upper-level information system.

10. The small billet tracking method according to claim 9, characterized in that, In step 5, the stack position sensing system identifies the number of rows and columns of the topmost blank in the stack position using an instance segmentation model, specifically including: A spatial index is constructed for the preprocessed point cloud, and distance thresholds, minimum cluster point count, and maximum cluster point count are set. By clustering, points with a spatial distance less than a threshold are grouped into the same cluster, and multiple point cloud clusters are output. Each cluster is traversed, the bounding box is calculated, and the cluster corresponding to the largest bounding box within the vehicle's position range is used as the clamp point cloud data. In load condition determination, if the normal vector of the point cloud in the jaw region is parallel to the normal vector of the adjacent point cloud below and the distance is within a preset threshold, it is determined to be overloaded.

Images