AGV fault positioning traction system and method based on visual recognition

Abstract

The application discloses an AGV fault positioning traction system and method based on visual identification, relates to the technical field of computer vision, and comprises the following steps: collecting scene images and task states of an AGV running area, performing visual analysis on an AGV vehicle body, a channel boundary, a cargo carrying contour and a traction interface, identifying the stagnation anomaly, attitude deviation and channel occupation state of a fault AGV, and constructing a fault pose spectrum; reconstructing a traction interface based on the fault pose spectrum, calculating a traction adaptation value by fusing the traction interface orientation, the vehicle body deflection angle, the channel residual width and the obstacle proximity relationship, and generating a traction execution corridor by combining particle swarm optimization, so that the traction equipment can complete visual correction hooking and dynamic traction transfer, and the reliability of fault AGV positioning, access and traction disposal is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to an AGV fault location and traction system and method based on visual recognition. Background Technology

[0002] In smart warehousing and flexible manufacturing scenarios, AGVs typically rely on scheduling and automated guidance control systems to perform transport tasks along preset channels. Affected by factors such as multi-vehicle collaboration, narrow channels, cargo offset, positioning errors, and vehicle posture disturbances, AGVs may experience abnormal stops, posture deviations, or channel obstruction during operation. If the control system cannot accurately obtain the actual position, vehicle posture, interface orientation, and space occupation range of a faulty AGV, subsequent path control, obstacle avoidance control, and fault handling control will lack reliable data, easily leading to continuous channel obstruction and localized scheduling blockages.

[0003] Existing AGV fault identification methods largely rely on vehicle status feedback, scheduling logs, or single location coordinates, making it difficult to combine operational scene images for unified analysis of vehicle deflection, cargo outline, channel boundaries, and interface orientation. When a faulty AGV is in an environment with skewed, outward-expanding cargo, or narrow channel conditions, generating a control path solely based on vehicle coordinates can easily lead to a mismatch between the approach direction and the actual interface posture, increasing access deviation and control risks. Furthermore, existing automatic guidance control methods in faulty vehicle transfer scenarios typically focus on pre-set path tracking or static obstacle avoidance, lacking collaborative modeling of the faulty AGV's occupied boundaries, remaining channel width, obstacle proximity relationships, and traction equipment motion constraints. This makes it difficult to optimize the execution path based on access deviation, path detour costs, corner restriction risks, and channel occupancy risks. Therefore, a visual recognition and automatic guidance collaborative solution for AGV fault handling control is urgently needed to improve the reliability of fault location, path generation, and dynamic traction control. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a visual recognition-based AGV fault location and traction system and method.

[0005] The technical solution to achieve the purpose of this invention is as follows:

[0006] On the one hand, the AGV fault location and traction method based on vision recognition includes the following steps:

[0007] Acquire scene images of the AGV operating area and AGV task status, perform visual analysis and temporal alignment of AGV body, channel boundary, cargo outline and traction interface to form operation perception data;

[0008] Based on operational perception data, stagnation anomalies, posture deviations, and channel occupancy states are identified. Combined with continuous frame displacement changes, faulty AGVs are determined and pose features are extracted to form a fault pose spectrum.

[0009] The traction interface is reconstructed based on the fault position spectrum. The traction interface orientation, vehicle body deflection angle, remaining channel width and obstacle proximity relationship are fused to calculate the traction adaptation value.

[0010] Based on the traction adaptation value, a traction constraint space is constructed. Particle swarm optimization is performed with access deviation, path detour cost, corner restriction risk and channel occupancy risk as optimization factors to generate a traction execution corridor.

[0011] The control traction equipment completes visual correction of the coupling posture and dynamic traction transfer based on the traction execution corridor, forming a fault traction handling result.

[0012] Furthermore, the formation of the fault pose spectrum includes:

[0013] Continuous time-series data sequences of AGVs are extracted from operational perception data, and the displacement of the vehicle center, changes in vehicle deflection angle, changes in traction interface orientation, channel occupancy area, and changes in task execution progress are calculated to form abnormal state time-series characteristics.

[0014] Based on the timing characteristics of abnormal states, stagnation anomalies, attitude deviations, and channel occupancy states are identified, and the faulty AGV is confirmed by combining the displacement changes and task state changes of consecutive frames before and after the occurrence of the abnormal state.

[0015] Extract the vehicle center, deflection angle, interface orientation, and occupied boundary of the faulty AGV. Perform accuracy verification and coordinate standardization on the extraction results to form a fault position and pose spectrum.

[0016] Further, the extraction of occupied boundaries includes:

[0017] The actual rectangle occupied by the vehicle body is generated based on the vehicle body center and deflection angle of the faulty AGV, and the actual rectangle occupied by the cargo is generated based on the cargo outline center and cargo deflection angle.

[0018] Merge the actual rectangle occupied by the vehicle body and the actual rectangle occupied by the cargo to obtain the occupancy boundary that includes the outer contour of the vehicle body and the outer extension of the cargo.

[0019] The stability of the occupied boundary is checked based on the pose changes of adjacent frames, and the checked occupied boundary is written into the fault pose spectrum.

[0020] Further calculations of the traction adaptation value include:

[0021] The vehicle center, deflection angle, traction interface center, traction interface orientation, and occupied boundary are extracted from the fault pose spectrum, and the channel boundary and obstacle contour are extracted from the operation perception data.

[0022] Establish a local coordinate system for the traction interface with the center of the traction interface and the orientation of the traction interface, transform the occupied boundary to the local coordinate system of the traction interface, and reconstruct the traction interface and traction constraint boundary;

[0023] The traction interface orientation, vehicle body deflection angle, remaining channel width, and obstacle proximity are quantified separately and then fused according to the degree of traction impact to obtain the traction adaptation value.

[0024] Furthermore, the reconstruction of the traction interface includes:

[0025] Based on the center and orientation of the traction interface, the local coordinate reference is determined, and the occupied boundary of the faulty AGV is mapped as a traction constraint polygon.

[0026] Within the local coordinate system of the traction interface, the traction interface, the vehicle body outline, the cargo outline, and the area adjacent to obstacles are expressed, forming a correspondence between the traction direction and the space occupation;

[0027] Based on the traction adaptation value, feasible traction directions are selected, and the optimal traction direction is determined to form a traction interface for constructing the traction constraint space.

[0028] Furthermore, the construction of the traction constraint space includes:

[0029] Extract the optimal traction direction, traction interface center, and traction constraint boundary from the traction interface, and determine the basic constraint parameters by combining the faulty AGV occupancy boundary, the current position of the traction equipment, the traction release area, and the channel topology information;

[0030] Based on the basic constraint parameters, a geometric constraint layer, an adaptive constraint layer, a kinematic constraint layer, and a safety constraint layer are constructed.

[0031] Low-adaptability areas, high-risk areas, areas occupied by obstacles, and areas outside the passage boundary are uniformly marked as impassable areas, forming a traction constraint space.

[0032] Furthermore, the generation of the traction execution corridor includes:

[0033] Candidate traction paths are constructed by using the sequence of path points from the current position of the traction equipment to the traction release area as particle codes.

[0034] The comprehensive traction cost of the candidate traction path is calculated based on access deviation, path detour cost, corner restriction risk, and channel occupancy risk.

[0035] Particle swarm optimization is performed within the traction constraint space to obtain the optimal traction path. After smoothing the optimal traction path, a traction execution corridor is generated.

[0036] Furthermore, the generation of the traction execution corridor also includes:

[0037] Using the optimal traction path as the centerline, calculate the allowable offsets on both sides of the centerline relative to the channel boundary and obstacles.

[0038] The left and right boundaries of the traction execution corridor are generated based on the allowed offset, and it is verified whether the traction execution corridor is located within the allowed passage area;

[0039] The verified traction execution corridor is output in a standardized manner according to the boundary vertex coordinates, centerline coordinates, path length, maximum turning angle, and comprehensive traction cost.

[0040] Further adjustments to the attachment posture include:

[0041] The traction equipment is controlled to approach the faulty AGV along the traction execution corridor, and an image of the traction interface area is acquired when it approaches the traction interface.

[0042] Key point detection is performed on the image of the traction interface area to obtain the corner point position of the traction interface, and then the center position and orientation of the traction interface relative to the traction equipment are obtained.

[0043] The attachment posture of the traction equipment is corrected based on the lateral, longitudinal, and angular deviations between the attachment mechanism and the traction interface.

[0044] Secondly, the AGV fault location and traction system based on vision recognition includes a running perception module, a pose positioning module, an adaptation calculation module, a corridor optimization module, and a traction control module:

[0045] The operation perception module acquires scene images of the AGV's operating area and the AGV's task status, performs visual analysis and temporal alignment of the AGV body, channel boundaries, cargo outline, and traction interface, and forms operation perception data.

[0046] The pose localization module identifies stagnation anomalies, posture deviations, and channel occupancy states based on operational perception data. It combines continuous frame displacement changes to determine faulty AGVs and extracts pose features to form a fault pose spectrum.

[0047] The adaptation calculation module reconstructs the traction interface based on the fault position spectrum, and integrates the traction interface orientation, vehicle body deflection angle, remaining channel width and obstacle proximity relationship to calculate the traction adaptation value.

[0048] The corridor optimization module constructs a traction constraint space based on the traction adaptation value, and performs particle swarm optimization with access deviation, path detour cost, corner restriction risk and channel occupancy risk as optimization factors to generate a traction execution corridor.

[0049] The traction control module controls the traction equipment to visually correct the coupling posture and dynamically traction transfer according to the traction execution corridor, thus forming a fault traction handling result.

[0050] Compared with the prior art, the advantages of this invention are as follows:

[0051] 1. By visually analyzing the AGV body, channel boundary, cargo outline, and traction interface, and combining the AGV task status for time-series alignment, the faulty AGV can be comprehensively identified from stagnation anomalies, posture deviations, and channel occupancy states. Furthermore, the vehicle body center, deflection angle, interface orientation, and occupancy boundary are extracted to form a fault pose spectrum, so that the fault location results not only include the vehicle position, but also the posture and space occupancy information directly related to traction access, which improves the accuracy of fault identification and pre-traction pose judgment in complex operating scenarios.

[0052] 2. Based on the fault position spectrum, the traction interface of the faulty AGV is reconstructed. The traction interface orientation, vehicle body deflection angle, remaining channel width, and obstacle proximity relationship are integrated into the traction adaptation value. The traction constraint space is constructed by combining the faulty AGV's occupied boundary, the current position of the traction equipment, and the traction release area. Then, particle swarm optimization is performed with access deviation, path detour cost, corner restriction risk, and channel occupancy risk as optimization factors to generate the traction execution corridor. This can improve the matching of traction access and the safety of the transfer process in narrow channels, posture deviation, and obstacle proximity scenarios. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the fault pose spectrum construction in this invention;

[0054] Figure 2 This is a schematic diagram of traction interface reconstruction and traction adaptation value calculation in this invention;

[0055] Figure 3 This is a schematic diagram illustrating the generation of the traction constraint space and traction execution corridor in this invention;

[0056] Figure 4 This is a diagram of the AGV fault location and traction system architecture based on vision recognition in this invention. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0058] Example 1

[0059] This invention discloses a visual recognition-based AGV fault location and traction method, comprising the following steps:

[0060] S1: Obtain scene images of the AGV operating area and AGV task status, perform visual analysis on the AGV body, channel boundary, cargo outline and traction interface in the scene image, and align them with the AGV task status in time sequence to form operation perception data.

[0061] S2: Based on the operational perception data, identify the AGV's stagnation anomaly, posture deviation and channel occupancy status, combine the continuous frame displacement changes to determine the faulty AGV and extract the pose features to form a fault pose spectrum. The pose features include the faulty AGV's vehicle center, deflection angle, interface orientation and occupancy boundary.

[0062] S3: Reconstruct the traction interface of the faulty AGV based on the fault position spectrum, and fuse the traction interface orientation, vehicle body deflection angle, remaining channel width and obstacle proximity relationship to calculate the traction adaptation value.

[0063] S4: Construct a traction constraint space based on traction adaptation value, faulty AGV occupancy boundary, current position of traction equipment and traction release area, and perform particle swarm optimization with access deviation, path detour cost, corner restriction risk and channel occupancy risk as optimization factors to generate traction execution corridor.

[0064] S5: Control the traction equipment to approach the faulty AGV along the traction execution corridor, correct the coupling posture based on the visual deviation between the traction interface and the traction equipment, and adjust the traction action according to the vehicle body sway, connection offset and channel occupancy changes during the traction transfer process to form the fault traction handling result.

[0065] In step S1, scene images of the AGV operating area and the AGV task status are acquired. Visual analysis is performed on the AGV body, channel boundaries, cargo outline, and traction interface in the scene image, and this is time-aligned with the AGV task status to form operational perception data, including:

[0066] S101: Synchronous acquisition of multi-source data.

[0067] A multi-view industrial camera array deployed on the ceiling of the AGV operating area synchronously acquires real-time scene images covering the entire operating channel. The camera array adopts a 2×N grid layout, with a preset horizontal spacing of 8m between adjacent cameras, a preset installation height of 3.5m, a single camera resolution of 1920×1080 pixels, a preset frame rate of 30fps, and a preset global exposure time of 1ms. All cameras achieve microsecond-level synchronous acquisition via hardware trigger signals, which are uniformly generated by an independent synchronization controller, with a synchronization accuracy better than ±10μs. After system deployment, the camera array undergoes a one-time global calibration with an accuracy better than ±3mm, and the calibration results are stored in the system parameter database.

[0068] Simultaneously, task status data of all online AGVs is acquired from the AGV scheduling system via ModbusTCP industrial Ethernet at a frequency of 10Hz. The industrial Ethernet transmission latency is less than 10ms, and a CRC32 checksum mechanism is used to ensure data transmission integrity. The task status data is transmitted in fixed-length frame format and includes 10 fields: AGV unique identifier, task number, target station number, current linear speed, angular velocity, travel direction indicator, cargo weight, cargo type code, battery percentage, and instruction execution progress percentage. The data type and length of each field are predefined. The system uses a circular data buffer with a capacity of 100 frames for temporary storage of the received task status data.

[0069] S102: Scene image normalization preprocessing.

[0070] Preprocessing operations are performed on the acquired real-time scene images, sequentially performing adaptive grayscale transformation, bilateral filtering for noise reduction, and gamma correction to generate a standardized scene image. The adaptive grayscale transformation uses a piecewise linear transformation method to map the image grayscale values ​​to the complete range of 0-255; the bilateral filtering uses a 5×5 convolution kernel with a preset spatial domain standard deviation of 15 and a preset grayscale domain standard deviation of 20, preserving image edge information while removing noise; the gamma correction coefficient is preset to 1.2 to adjust the overall brightness of the image.

[0071] Contrast in low-light areas of the image is enhanced by limiting contrast-adaptive histogram equalization. The block size is preset to 8×8 pixels, and the cropping limit is preset to 0.02 to prevent over-enhancement and noise. The edge features of the AGV body contour, channel boundaries, and traction interface are strengthened using the Sobel edge enhancement operator, with both horizontal and vertical gradient kernels using 3×3 matrices. After preprocessing, abnormal image detection is performed. The average grayscale value of the image is calculated, and images with an average grayscale value below 30 or above 220 are identified as abnormal. The abnormal image is replaced by the average result of two adjacent frames. This threshold covers both overly dark and overexposed abnormal image scenarios, effectively identifying extreme acquisition conditions such as light source failure and direct strong light.

[0072] S103: Multi-target instance segmentation and basic feature extraction.

[0073] Standardized scene images are input into a pre-trained improved YOLOv8-seg multi-object instance segmentation network, which outputs AGV body instance masks, channel boundary instance masks, cargo outline instance masks, and traction interface instance masks. For each instance mask, basic geometric features are extracted, including the minimum bounding rectangle, center point coordinates, rotation angle, and pixel area. The rotation angle is the angle between the longer side of the minimum bounding rectangle and the horizontal axis of the image, ranging from -90° to 90°.

[0074] The backbone network of the improved YOLOv8-seg multi-object instance segmentation network adopts a cross-stage local network fusion structure (C2f). This structure enhances feature representation capabilities through parallel branch design and includes five feature extraction stages at different scales, with output channel numbers of 64, 128, 256, 512, and 1024, respectively. Each cross-stage local network fusion structure (C2f) module consists of one 1×1 convolutional layer, two parallel branches, and one concatenation layer. Each branch contains a varying number of bottleneck modules, with the number of bottleneck modules in the five stages being 3, 6, 6, 3, and 3, respectively. The neck network adopts a path aggregation network-feature pyramid network (PAN-FPN) structure and embeds a convolutional block attention module (CBAM). A CBAM is inserted after each feature fusion node, simultaneously introducing channel attention and spatial attention mechanisms during feature fusion to improve the detection and segmentation capabilities for small targets and weak edge features. The segmentation head adopts a decoupled head structure, separating the classification task and the segmentation task. The classification branch outputs the confidence scores of the four categories, and the segmentation branch outputs the instance mask with the same size as the input image.

[0075] The training dataset contains 12,000 annotated AGV operation scene images, covering AGV images under different lighting conditions, different loading states, and different shooting angles. The training set comprises 80%, the validation set 10%, and the test set 10%. Multiple data augmentation strategies were employed during training, including random horizontal flipping, random vertical flipping, random brightness adjustment, random contrast adjustment, and mosaic enhancement. The training batch size was preset to 16, the initial learning rate to 0.01, a cosine annealing learning rate scheduling strategy, a maximum number of iterations to 300, and the AdamW optimizer with a weight decay coefficient of 0.0005 and a momentum of 0.937. After training, the model achieved a mean accuracy at 0.5 (mAP@0.5) of 98.7% and a mask segmentation intersection-over-union (IoU) of 96.2% on the test set.

[0076] S104: Geometric feature calculation and coordinate transformation.

[0077] For the AGV vehicle body instance mask, the Canny edge detection algorithm is used to extract the vehicle body edge feature points. The Canny algorithm has a preset low threshold of 50 and a preset high threshold of 150. The 1:3 ratio of high to low thresholds is a classic configuration for industrial edge detection, which can effectively filter false edges while preserving the real vehicle body outline and adapt to the grayscale distribution characteristics of the preprocessed image. The least squares method is used to perform rectangle fitting on the extracted edge feature points to obtain the vehicle body outline rectangle. The scaling factor between the actual vehicle body size and the image pixel size is calculated. The scaling factor is calculated by dividing the actual vehicle body length by the pixel length of the vehicle body outline rectangle, where the actual vehicle body length is the pre-stored AGV model parameter, preset to 1200mm.

[0078] By combining the camera intrinsic parameter matrix and the pre-calibrated extrinsic parameter matrix, the coordinates of the vehicle's center point in the image coordinate system are converted to absolute coordinates in the world coordinate system through perspective transformation. The coordinate transformation process consists of two steps: first, converting the image coordinate system coordinates to the camera coordinate system coordinates, and then converting the camera coordinate system coordinates to the world coordinate system coordinates, with a transformation accuracy better than ±5mm. For the channel boundary instance mask, the Hough linear transform is used to fit the equations of the straight lines on both sides of the channel. The distance resolution of the Hough linear transform is preset to 1 pixel, the angle resolution to 1°, and the voting threshold to 100. The actual channel width and the equation of the channel centerline are calculated. The channel width is calculated as the vertical distance between the two boundary straight lines.

[0079] For the cargo outline instance mask, extract the top and side outlines of the cargo, and calculate the cargo volume and its offset relative to the AGV body. The cargo offset is calculated as the Euclidean distance between the center point of the cargo outline and the center point of the AGV body on the horizontal plane. For the traction interface instance mask, extract the interface center coordinates and the interface axis direction vector, and calculate the relative position and orientation angle of the traction interface relative to the center of the AGV body. The orientation angle is calculated as the angle between the interface axis direction vector and the AGV body's forward direction vector. The cosine of the angle is calculated through the vector dot product, and the direction of the angle is determined through the vector cross product, with a value ranging from -180° to 180°.

[0080] S105: Time-series alignment and correlation of multi-source data.

[0081] A unified global time base based on a Network Time Protocol (NTP) server is established, and all devices synchronize their time with the NTP server with a synchronization accuracy better than ±1ms. The timestamps for scene image acquisition and AGV task status data generation are uniformly converted to Unix timestamp format with millisecond-level accuracy.

[0082] For each scene image acquisition time, the two closest time points to the acquisition time are found in the circular data buffer, and the timestamp deviation is calculated. If both are less than a preset time threshold of 50ms, a timing alignment operation is performed. The 50ms threshold matches the frame interval of a 30fps image, ensuring that the time deviation between the interpolated and aligned status data and the visual data does not affect the fault determination result. For continuous task status data, including current linear velocity, angular velocity, battery percentage, and instruction execution progress percentage, the corresponding value at time t is calculated based on the values ​​of two adjacent time points. For discrete task status data, including the AGV unique identifier, task number, target station number, driving direction identifier, and cargo type code, the value corresponding to the closest time point to t is taken.

[0083] The task status data, which are aligned with the AGV vehicle position, deflection angle, cargo status, traction interface pose and time in the world coordinate system, are associated one by one to form a single AGV spatiotemporal associated data unit containing timestamp, unique AGV identifier, spatial pose information, cargo information and task information.

[0084] S106: Data consistency verification and structured organization.

[0085] Multi-dimensional consistency checks are performed on all spatiotemporal correlated data units of individual AGVs. First, a position consistency check is performed, calculating the shortest distance between the actual AGV position and the planned path of the AGV scheduling system. If the shortest distance exceeds a preset distance threshold of 100mm, the data unit is determined to be an abnormal data unit. Second, a speed consistency check is performed, calculating the displacement of the AGV in two adjacent frames and dividing it by the time interval to obtain the actual speed. If the deviation between the actual speed and the speed in the task status data exceeds a preset speed threshold of 0.1m / s, the data unit is determined to be an abnormal data unit. Finally, a task status logic check is performed. If the AGV is in a loaded state but the load weight is 0, or in an empty state but the load weight is greater than 0, the data unit is determined to be an abnormal data unit. The 0.1m / s deviation threshold matches the fluctuation range of the AGV's normal driving speed, effectively identifying speed data anomalies.

[0086] All data units deemed abnormal were removed, and the timestamp, unique AGV identifier, and abnormality type of the abnormal data were recorded in the system abnormality log. The verified spatiotemporal correlation data units of individual AGVs were organized according to time sequence and spatial location to construct structured operational perception data containing global channel topology information, real-time pose information of all AGVs, cargo attribute information, and task execution information. The operational perception data was stored in JSON format, with an update frequency consistent with the camera acquisition frequency of 30Hz. The global channel topology information included channel number, channel start coordinates, channel end coordinates, channel width, and channel type; the real-time pose information of the AGVs included the unique AGV identifier, world coordinate system x-coordinate, y-coordinate, deflection angle, traction interface center coordinates, and traction interface orientation angle.

[0087] In step S2, based on the operational perception data, the AGV's stagnation anomaly, attitude deviation, and channel occupancy status are identified. Combined with continuous frame displacement changes, the faulty AGV is determined, and the vehicle body center, deflection angle, interface orientation, and occupancy boundary of the faulty AGV are extracted to form a fault pose spectrum, including:

[0088] S201: Extraction of temporal features of abnormal states.

[0089] Continuous time-series data sequences of all online AGVs are extracted from structured operational sensing data. The time window length of the time-series data sequence is preset to 3 seconds, containing 90 consecutive frames of spatiotemporal correlation data units of a single AGV, numbered in chronological order. .

[0090] For each AGV, frame-by-frame feature calculation is performed on the time-series data sequence to generate a five-dimensional time-series feature vector. The sequence of vehicle body center displacements. The Frame elements The calculation formula is:

[0091] ,

[0092] in, For the first The world coordinates of the AGV's body center. For the first The world coordinates of the AGV body center.

[0093] Vehicle body deflection angle change sequence The Frame elements ,in For the first Frame AGV body deflection angle; traction interface orientation change sequence The Frame elements ,in For the first Frame traction interface orientation angle; channel occupancy area sequence The Frame elements For the first The overlapping area between the frame AGV's occupancy boundary and the channel boundary is calculated using a polygon intersection area algorithm; the task execution progress change sequence. The Frame elements ,in For the first Percentage of AGV task execution progress.

[0094] S202: Multi-dimensional abnormal state identification.

[0095] Identify AGV stagnation / abnormal states. Calculate the cumulative displacement of the AGV within the time window. and average speed The calculation formulas are as follows:

[0096] ,

[0097] ,

[0098] When the condition is met Preset displacement threshold 5mm and and If the AGV does not receive a stop command from the scheduling system, it is determined that the AGV is in a stagnation abnormality, and a stagnation abnormality flag is set. ,otherwise .

[0099] Identify the AGV's attitude deviation state. Calculate the... The deviation between the AGV body deflection angle and the corresponding planned path tangent angle. The calculation formula is:

[0100] ,

[0101] in, For the first The tangent angle of the planned path at the location of the AGV is calculated by linear interpolation using the path node coordinates provided by the AGV scheduling system.

[0102] Within the statistical time window The number of consecutive frames exceeding a preset angle threshold of 15° ,when When the preset frame rate threshold is 30, the AGV is determined to have an abnormal attitude deviation, and an abnormal attitude deviation flag is set. ,otherwise .

[0103] Identify the AGV's channel occupancy status. Calculate the... The overlapping area between the frame AGV's occupied boundary and the left boundary of the channel. and the overlapping area with the right boundary of the channel The steps for calculating the overlapping area are as follows:

[0104] Channel boundary region definition: The left boundary line of the channel, obtained by fitting through Hough linear transform, is extended outward by a preset extension threshold of 0.5m to form a left boundary rectangle; the right boundary line of the channel is also extended outward by a preset extension threshold of 0.5m to form a right boundary rectangle. The 0.5m extension width covers the safety margin of the channel edge, accurately identifying abnormal states where AGVs overstep their boundaries and encroach on the channel. The longitudinal range of the two boundary rectangles covers the entire length of the channel segment currently occupied by the AGV.

[0105] Polygon intersection calculation: The Sutherland-Hodgman polygon clipping algorithm is used to calculate the intersection polygons of the AGV's boundary polygon and the left boundary rectangle, as well as the intersection polygons of the AGV's boundary polygon and the right boundary rectangle. The specific execution steps of the Sutherland-Hodgman polygon clipping algorithm are as follows:

[0106] Input the polygon to be clipped (the AGV occupies the boundary polygon, with vertices arranged in clockwise order) and the clipping window (the left / right boundary rectangle, with vertices arranged in clockwise order), and initialize the output polygon vertex list to be the vertex list of the polygon to be clipped.

[0107] The output polygon vertex list is clipped sequentially using the four edges of the clipping window. The clipping process for each edge is as follows:

[0108] Initialize the temporary vertex list after clipping the current edge to empty; take the last vertex of the output polygon vertex list as the previous vertex. Iterate through and output each vertex in the list of polygon vertices as the current vertex. Determine the vertex and Relative to the current clipping edge position: if the vertex is inside or on the clipping edge, it is determined as an internal point; if the vertex is outside the clipping edge, it is determined as an external point; if... For interior points and If it is an interior point, then... Add to the temporary vertex list; if For interior points and If it is an external point, then calculate the line segment. Intersection with the current cropped edge , intersection Add to the temporary vertex list; if external point and If it is an interior point, then calculate the line segment. Intersection with the current cropped edge , intersection and vertex Add them sequentially to the temporary vertex list; if external point and If the vertex is an external vertex, no vertex is added to the temporary vertex list; after traversal, the output polygon vertex list is updated to the temporary vertex list.

[0109] Once all four sides of the clipping window have been processed, the output polygon vertex list is the list of vertices of the intersection polygon between the AGV's occupied boundary polygon and the corresponding boundary rectangle.

[0110] Method for calculating the intersection point of a line segment and the trimmed edge: Let the equation of the straight line of the trimmed edge be... line segment The coordinates of the two endpoints are respectively and Then the intersection point The calculation formula is:

[0111] ,

[0112] ,

[0113] ,

[0114] in, The intersection point is on the line segment The parameters on, when At that time, the intersection point is on the line segment superior.

[0115] Intersection Area Calculation: The area of ​​the intersecting polygons is calculated using the shoelace formula. The shoelace formula expression is:

[0116] ,

[0117] in, The area of ​​the polygon. The number of vertices of the polygon. For polygon number 1 The coordinates of each vertex are specified. .

[0118] The area of ​​the left intersection polygon is calculated as follows: The area of ​​the intersection polygon on the right is... .

[0119] When the condition is met Preset area threshold or Preset area threshold or Preset area threshold If the AGV is found to have an abnormal channel occupancy, a channel occupancy abnormality flag is set. ,otherwise The boundary occupied by the AGV is the union of the smallest bounding rectangle of the AGV body outline and the cargo outline.

[0120] S203: Comprehensive judgment of faulty AGV.

[0121] For AGVs with at least one abnormal state, i.e. Verify the displacement change across consecutive frames. Extract the vehicle center coordinates of the AGV for 10 frames before and after the occurrence of the abnormal state, and calculate the average displacement over the 10 consecutive frames. The calculation formula is:

[0122] ,

[0123] in, This is the frame number where the abnormal state first appears.

[0124] when When the preset displacement threshold is 3mm, excluding non-fault situations such as temporary deceleration and normal docking, the AGV is initially determined to be a faulty AGV. A secondary verification of the task status is performed on the initially determined faulty AGV. If the task execution progress remains unchanged for 10 consecutive frames and the AGV's battery percentage is greater than the preset battery threshold of 20%, the AGV is finally confirmed as a faulty AGV. The unique identifier, anomaly type combination, and fault occurrence timestamp of all finally confirmed faulty AGVs are recorded in the system fault log.

[0125] S204: Refined extraction of pose features of faulty AGVs.

[0126] Extract the spatiotemporal correlation data unit of the faulty AGV at the moment of fault confirmation, and obtain the world coordinates of the AGV body center at that moment. Vehicle body deflection angle and the orientation angle of the traction interface The vehicle body deflection angle The AGV vehicle's forward direction relative to the world coordinate system The angle between the positive axes has a range of values. to ; Traction interface orientation angle The traction interface axis direction and the world coordinate system The angle between the positive axes has a range of values. to .

[0127] Generate the coordinates of the four vertices of the rectangle actually occupied by the vehicle body. First, define the relative coordinates of the four vertices of the vehicle body rectangle with respect to the center of the vehicle body in the unrotated state, as follows: , , , ,in, This refers to the actual length of the vehicle body. This represents the actual width of the vehicle body. Each relative coordinate is multiplied by a rotation matrix to perform an angular rotation. The rotation matrix is:

[0128] ,

[0129] After rotation, add the world coordinates of the vehicle's center to the coordinates of each vertex. This gives the coordinates of the four vertices of the rectangle actually occupied by the vehicle body in the world coordinate system.

[0130] Generate the coordinates of the four vertices of the rectangle actually occupied by the cargo. First, define the relative coordinates of the four vertices of the unrotated cargo rectangle with respect to the center of the cargo outline, as follows: , , , ,in, This is the actual length of the cargo. This represents the actual width of the cargo. Each relative coordinate is multiplied by a rotation matrix to perform an angular rotation, the rotation angle being the cargo deflection angle. After rotation, add the world coordinates of the cargo outline center to the coordinates of each vertex. This gives the coordinates of the four vertices of the rectangle actually occupied by the cargo in the world coordinate system.

[0131] The union of the rectangle actually occupied by the vehicle body and the rectangle actually occupied by the cargo is taken as the final occupancy boundary of the faulty AGV. The occupancy boundary is represented by the world coordinates of the four vertices arranged in clockwise order.

[0132] The extracted pose features are subjected to accuracy verification. The Euclidean distance between the center coordinates of the faulty AGV in the current frame and the weighted average coordinates of the three adjacent frames is calculated, where the weight of the current frame is 0.6, and the weights of the previous and next frames are each 0.2. When the distance is less than a preset accuracy threshold of 10mm, the pose feature is confirmed to be valid; when the distance is greater than or equal to the preset accuracy threshold of 10mm, the weighted average of the three adjacent frames is used as the final pose feature.

[0133] S205: Fault Pose Spectrum Construction and Standardization.

[0134] The unique identifier, abnormality type combination, and vehicle center coordinates of the faulty AGV. Vehicle body deflection angle traction interface orientation angle Fault occurrence timestamp The coordinates of occupied boundary vertices are integrated to construct the initial fault pose spectrum.

[0135] The initial fault pose spectrum was standardized. All coordinates were converted to absolute coordinates in the world coordinate system, retaining three decimal places; all angles were also standardized to... to Values ​​within the range are rounded to one decimal place; the clockwise sorting of the boundary vertex coordinates is reconfirmed. The standardized fault pose spectrum is stored in JSON format, containing 12 standardized fields. The data structure is fully compatible with the operational awareness data and can be directly used for subsequent traction interface reconstruction and traction path planning.

[0136] refer to Figure 1 , Figure 1 A schematic diagram for constructing the fault pose spectrum.

[0137] Figure 1 This paper demonstrates the process of identifying faulty AGVs, normal AGVs, channel boundaries, cargo outlines, and traction interfaces through visual analysis, and extracting vehicle center, deflection angle, interface orientation, and occupancy boundary by combining stagnation anomalies, attitude deviations, and channel occupancy states, thereby constructing a fault pose spectrum.

[0138] It should be noted that, Figure 1 This is for illustrative purposes only, intended to help understand the process structure and data organization, and does not represent the precise working state in actual operation.

[0139] In step S3, the traction interface of the faulty AGV is reconstructed based on the fault pose spectrum. The traction interface orientation, vehicle body deflection angle, remaining channel width, and obstacle proximity are fused together to calculate the traction adaptation value, including:

[0140] S301: Extraction of basic parameters of traction interface.

[0141] Extracting the core pose parameters of the faulty AGV from the standardized fault pose spectrum, including the world coordinates of the vehicle center. Vehicle body deflection angle traction interface center world coordinates traction interface orientation angle and the coordinates of the occupied boundary vertices .

[0142] Extract environmental parameters of the channel where the faulty AGV is located from structured operation perception data, including the straight line equation of the channel centerline and the actual width of the channel. The equations of the straight lines on the left and right boundaries of the passage are obtained. Using the center of the faulty AGV as the center, all obstacles within a preset scanning threshold of 5m are scanned to obtain the contour vertex coordinates and type identifiers of the obstacles, distinguishing between static and dynamic obstacles.

[0143] S302: Geometric reconstruction of the traction interface of a faulty AGV.

[0144] Establish a local coordinate system for the traction interface, with the center of the traction interface as the reference point. Using the origin of the coordinate system and the angle of the traction interface as the reference point. Let x be the positive x-axis, and let y be the positive y-axis, perpendicular to the x-axis and pointing to the left side of the AGV body. Construct a transformation matrix between the local coordinate system of the traction interface and the world coordinate system. The transformation formula is:

[0145] ,

[0146] in, Coordinates in the world coordinate system These are the coordinates in the local coordinate system of the traction interface.

[0147] The geometric model of the traction interface is reconstructed based on the transformation matrix. The traction interface is a standard rectangular structure with a preset interface length of 100mm and a preset interface width of 50mm. This size is the standard specification for general traction coupling interfaces of industrial AGVs and has wide industry adaptability. The vehicle body traction constraint boundary is reconstructed, and the boundary occupied by the faulty AGV is transformed to the local coordinate system of the traction interface, forming a complete traction constraint polygon that includes the vehicle body and the cargo.

[0148] S303: Quantification of multi-dimensional influencing factors.

[0149] The four dimensions of traction interface orientation, vehicle body deflection angle, remaining channel width, and obstacle proximity are quantified to generate values ​​ranging from [value range missing]. The standardized influence factor is such that the larger the value, the better the adaptability of this dimension to traction operations.

[0150] Traction Interface Orientation Deviation Factor Quantization. Calculating the approach direction of the traction equipment. Orientation of traction interface absolute deviation between The calculation formula is:

[0151] ,

[0152] when At that time, take The formula for calculating the traction interface orientation deviation factor is:

[0153] ,

[0154] Factors affecting vehicle body deflection angle Quantization. Calculate the vehicle body deflection angle. Angle with the center line of the channel absolute deviation between The calculation formula is:

[0155] ,

[0156] when At that time, take The formula for calculating the influencing factor of vehicle body deflection angle is:

[0157] ,

[0158] when At that time, take .

[0159] Channel Remaining Width Influence Factor Quantization. Calculate the remaining width on both sides of the passage after the faulty AGV occupies it, including the remaining width on the left side. Remaining width on the right ,in, This is the vertical distance from the center of the AGV vehicle to the left edge of the passage. This is the vertical distance from the center of the AGV to the right edge of the passage. The smaller of the remaining widths on both sides is taken as the effective remaining width. The formula for calculating the channel remaining width impact factor is:

[0160] ,

[0161] in, The minimum channel width required for the traction equipment to perform traction operations is preset to 1200mm.

[0162] Barrier proximity influence factor Quantization. Calculate the distance to the nearest obstacle on the traction path from the faulty AGV. This distance is obtained by calculating the shortest Euclidean distance between the straight line of the traction direction and the polygonal outlines of all obstacles, and taking the minimum value as the nearest obstacle distance. The formula for calculating the obstacle proximity influence factor is:

[0163] ,

[0164] in, The safe distance threshold for traction operations is preset to 1000mm. When At that time, take This indicates that there is a collision risk in that direction, and towing is prohibited; among them, 200mm is the minimum safe clearance limit for towing equipment. Below this value, there is a clear collision risk, so it is directly determined to be infeasible.

[0165] S304: Weighted fusion calculation of traction adaptation value.

[0166] The four standardized influence factors were merged into a comprehensive traction fit value using a weighted linear fusion method. The calculation formula is:

[0167] ,

[0168] in, , , , These are the weight coefficients of the four influencing factors, and they satisfy... The preset weighting coefficients are as follows: , , , The weighting is based on the core logic of the traction operation, which is that the orientation of the traction interface directly determines the success rate of the hook-up, making it the most critical factor. Therefore, it is given the highest weight of 0.4. The vehicle body deflection angle, the remaining width of the passage, and the proximity of obstacles have a similar impact on the safety and difficulty of traction, so each is given an equal weight of 0.2.

[0169] Traction Adaptation Value The range of values ​​is A higher value indicates better overall adaptability in the traction direction, resulting in higher safety and lower difficulty in traction operations. When If the traction direction is deemed infeasible, it should be excluded.

[0170] S305: Determination of optimal traction direction and output of traction interface.

[0171] In the local coordinate system of the traction interface Within the range, all possible traction directions are traversed in a preset angle step of 1°, and the traction adaptation value corresponding to each direction is calculated. All feasible directions with traction adaptation values ​​greater than or equal to a preset adaptation threshold of 0.6 are selected, and the direction with the largest traction adaptation value is chosen as the optimal traction direction.

[0172] The final traction interface is reconstructed based on the optimal traction direction. The output includes the center coordinates of the traction interface, the orientation angle of the traction interface, the optimal traction direction, the traction constraint boundary polygon, the distribution of traction adaptation values ​​in each direction, and the infeasible traction region. The reconstructed traction interface is stored in JSON format, and its data structure is fully compatible with the fault pose spectrum, allowing it to be directly used for subsequent traction constraint space construction.

[0173] refer to Figure 2 , Figure 2 This is a schematic diagram of the geometric reconstruction and traction adaptation calculation of the traction interface.

[0174] Figure 2 The process is illustrated by establishing local geometric relationships based on the center and orientation of the traction interface, reconstructing the traction interface by combining the occupied boundary polygon, the remaining width of the channel, the adjacent area of ​​the obstacle, and the candidate traction direction, and generating traction adaptation values ​​and a set of feasible traction directions through the traction interface orientation factor, the vehicle body deflection angle factor, the remaining width of the channel factor, and the obstacle proximity factor.

[0175] It should be noted that, Figure 2 This is for illustrative purposes only, intended to help understand the process structure and data organization, and does not represent the precise working state in actual operation.

[0176] In step S4, a traction constraint space is constructed based on the traction adaptation value, the faulty AGV occupancy boundary, the current position of the traction equipment, and the traction release area. Particle swarm optimization is then performed using access deviation, path detour cost, corner restriction risk, and channel occupancy risk as optimization factors to generate the traction execution corridor, including:

[0177] S401: Extraction of basic parameters of traction constraint space.

[0178] Extract the optimal traction direction from the reconstructed traction interface. traction interface center coordinates Data on the traction constraint boundary polygon and the distribution of traction adaptation values ​​in all directions. The coordinates of the occupied boundary vertices of the faulty AGV are extracted from the standardized fault pose spectrum. and vehicle body deflection angle .

[0179] Obtain the current pose parameters of the traction equipment from the traction equipment control system, including the world coordinates of the traction equipment center. traction equipment deflection angle The system also includes the geometric dimensions of the traction equipment, with a preset length of 1500mm and a preset width of 800mm. These dimensions match the standard specifications of small traction AGVs, allowing for flexible movement within narrow warehouse aisles and adapting to traction failure scenarios. Preset traction release area parameters are extracted from the system task database. The traction release area is a rectangular region, defined by the coordinates of its top-left vertex. and the coordinates of the bottom right vertex definition.

[0180] Global channel topology information and real-time obstacle distribution information are extracted from structured operational perception data, including the boundary line equations of all channels, channel widths, and the contour vertex coordinates of all obstacles within a preset scanning threshold of 5m.

[0181] S402: Construction of multi-dimensional traction constraint space.

[0182] A four-dimensional traction constraint space is constructed, consisting of a geometric constraint layer, an adaptability constraint layer, a kinematic constraint layer, and a safety constraint layer. All constraints are uniformly expressed in the world coordinate system.

[0183] Geometric constraint layer construction. Define the set of impassable regions. This includes the area inside the boundary occupied by the faulty AGV, the area inside the outline of all obstacles, and the area outside the channel boundary. Define the area that must be reached. Define the towing release area. Define the permitted passage area. Subtract the set of impassable regions from the global passage region. The geometric constraints are expressed as: It belongs to the traction path. The intersection with the traction path is not empty.

[0184] Adaptive constraint layer construction. An adaptive constraint field is constructed based on the omnidirectional traction adaptation value distribution data. For any point in the world coordinate system... Its adaptability constraint value This is the traction adaptation value corresponding to the direction pointing from this point to the center of the traction interface. The adaptation constraint is expressed as:

[0185] ,

[0186] For regions with a fit constraint value less than 0.3, they are marked as low-fitness, impassable regions and added to the impassable region set. .

[0187] Kinematic constraint layer construction. Based on the kinematic characteristics of the traction equipment, kinematic constraints are constructed. The maximum steering angle of the traction equipment is preset to... The minimum turning radius is preset to 1500mm, and the maximum driving speed is preset to 0.5m / s. The kinematic constraints require that the radius of curvature of the traction path is greater than or equal to the minimum turning radius at all points, and that the turning angle between adjacent path segments does not exceed the maximum steering angle.

[0188] Constructing a safety constraint layer. This involves building a safety distance constraint field for any point in the world coordinate system. Its distance to the nearest obstacle is The safety constraints are expressed as follows:

[0189] ,

[0190] Areas with a safe distance of less than 500mm are marked as high-risk, impassable areas and added to the impassable area set. .

[0191] S403: Construction of Particle Swarm Optimization Model.

[0192] A particle swarm optimization model is constructed with the goal of minimizing the overall traction cost. Each particle represents a candidate traction path, and a path point sequence encoding method is used. The number of path points is preset to 10, which are evenly distributed between the current position of the traction equipment and the traction release area.

[0193] Optimize the objective function definition. Consider the overall traction cost. Access deviation cost Detour Cost Risk of limited turning angle and the risk of channel blockage The weighted fusion is obtained, and the calculation formula is:

[0194] ,

[0195] in, , , , Let be the weight coefficient, and satisfy... The preset weighting coefficients are as follows: , , , The weighting is based on the priority of the traction operation: the connection accuracy directly determines the success rate of the connection and is given the highest weight; the path length affects the traction efficiency and is given the second highest weight; the cornering limit and the channel occupancy affect the traction safety and are given the same weight.

[0196] Quantitative calculation of each optimization factor:

[0197] Access deviation cost Calculate the positional and angular deviations between the end point of the traction path and the optimal traction engagement point. The calculation formula is as follows:

[0198] ,

[0199] in, Let Euclidean distance be the distance between the end point of the path and the center of the traction interface. The maximum permissible connection deviation is preset to 200mm. This represents the angular deviation between the path endpoint and the optimal traction direction.

[0200] Path detour cost Calculate the ratio of the candidate path length to the theoretical shortest path length using the following formula:

[0201] ,

[0202] in, The total length of the candidate path is obtained by calculating and summing the Euclidean distances between all adjacent path points in the candidate path in turn. Given the theoretical shortest path length from the current position of the traction equipment to the traction release area, the collision-free shortest path length is calculated using the A* algorithm within the permitted passage area.

[0203] Risk of limited cornering Calculate the weighted sum of the turning angles of all adjacent path segments in the candidate path. The formula is:

[0204] ,

[0205] in, The number of path points. For the first The corners at each path point.

[0206] Channel congestion risk Calculate the maximum proportion of a candidate path where both the traction device and the faulty AGV simultaneously occupy the lane. The calculation formula is as follows:

[0207] ,

[0208] in, For the first The total width occupied by the traction equipment and the faulty AGV at each path point. This represents the channel width at the corresponding position.

[0209] Particle swarm optimization (PSO) algorithm parameter settings. The particle swarm size is preset to 50, the maximum number of iterations is preset to 100, and the inertia weight... The individual learning factor decreases linearly from 0.9 to 0.4. Global learning factor .

[0210] S404: Optimal traction path solution and smoothing process.

[0211] Initialize the particle swarm, with each particle's initial position being a randomly generated sequence of path points satisfying the constraints. Execute an iterative particle swarm optimization process, calculating the combined traction cost of all particles in each iteration and updating the individual optimal position and the global optimal position.

[0212] When the number of iterations reaches the maximum number of iterations or the change in the global optimal cost after 20 consecutive iterations is less than 1... When the iteration ends, the path corresponding to the globally optimal position is output as the initial optimal traction path.

[0213] The initial optimal traction path was smoothed using cubic B-spline curves to eliminate sharp corners and ensure continuous path curvature. The number of path points increased to 50 after smoothing, and the path accuracy was better than ±10mm.

[0214] S405: Traction Execution Corridor Generation and Standardized Output.

[0215] Based on the smoothed optimal traction path, a traction execution corridor with variable width is generated. For each point on the path, the maximum allowable offset to the left and right of that point is calculated. The maximum offset to the left is the distance from that point to the left boundary of the corridor minus the safety distance of 500mm, and the maximum offset to the right is the distance from that point to the right boundary of the corridor minus the safety distance of 500mm.

[0216] Using the optimal traction path as the centerline and the maximum allowable offset on both sides as half-width, the left and right boundaries of the traction execution corridor are generated. The traction execution corridor is represented by the coordinates of the left and right boundary vertices arranged in sequence, with the coordinates of the left boundary vertex being... The coordinates of the right boundary vertex are .

[0217] The generated traction execution corridor is validated to ensure it is entirely within the permitted passage area and that all constraints are met at any point within the corridor. After successful validation, the traction execution corridor parameters are standardized and stored in JSON format, including corridor boundary vertex coordinates, centerline coordinates, path length, maximum turning angle, and comprehensive traction cost fields. The data structure is fully compatible with the traction interface and can be directly used for subsequent traction equipment control.

[0218] refer to Figure 3 , Figure 3 A schematic diagram of the construction of the traction constraint space and the generation of the traction execution corridor.

[0219] Figure 3 This paper illustrates the process of constructing a traction constraint space based on geometric constraint layers, adaptation constraint layers, kinematic constraint layers, and safety constraint layers under constraints such as the boundary of the faulty AGV, the current position of the traction equipment, obstacles, the traction release area, and low adaptability area, high risk area, and impassable area. The process of generating the optimal traction path and traction execution corridor is then achieved through particle swarm optimization by considering access deviation, path detour cost, corner restriction risk, and channel occupancy risk.

[0220] It should be noted that, Figure 3 This is for illustrative purposes only, intended to help understand the process structure and data organization, and does not represent the precise working state in actual operation.

[0221] In step S5, the traction equipment approaches the faulty AGV along the traction execution corridor, corrects the coupling posture based on the visual deviation between the traction interface and the traction equipment, and adjusts the traction action according to vehicle body sway, connection offset, and changes in channel occupancy during the traction transfer process, forming the fault traction handling result, including:

[0222] S501: Traction equipment path tracking and proximity control.

[0223] The centerline coordinate sequence and left and right boundary coordinate sequences of the traction execution corridor are extracted from the standardized traction execution corridor data. Starting from the current position of the traction equipment, a pure tracking algorithm is used to achieve high-precision path tracking of the traction execution corridor.

[0224] The control cycle of the pure tracking algorithm is preset to 100ms. This control cycle matches the motion control response frequency of industrial AGVs, ensuring real-time tracking and control stability. Within each control cycle, the following operations are performed: calculating the lateral deviation between the current position of the traction device and the corridor centerline. and heading deviation Among them, lateral deviation The shortest perpendicular distance from the current center coordinates of the traction equipment to the nearest line segment of the corridor centerline is calculated using the distance formula from a point to a line segment; heading deviation. The value is the difference between the current heading angle of the traction equipment and the tangent direction angle of the corridor centerline at the nearest point. After angle normalization, the value range is [value missing]. to Based on the current speed of the traction equipment and aiming distance The formula for determining the aiming point and calculating the aiming distance is as follows: The speed proportionality coefficient The preset time is 1.0s; calculate the target steering angle of the traction equipment. The calculation formula is:

[0225] ,

[0226] in, The wheelbase of the traction equipment is preset to 1000mm. It is the angle between the heading of the traction equipment and the line connecting the aiming point.

[0227] When the distance between the traction equipment and the center of the traction interface is less than the preset approach threshold of 2m, the traction equipment speed is reduced from 0.5m / s to 0.1m / s, entering the low-speed precision approach phase. During the low-speed approach phase, the path tracking accuracy is controlled within ±20mm, and the heading angle control accuracy is controlled within ±1°.

[0228] S502: Visual recognition and attachment posture correction of traction interface.

[0229] Once the traction equipment enters the low-speed precision approach phase, the industrial camera at the front end of the traction equipment is activated to acquire images of the traction interface area of ​​the faulty AGV in real time. The acquired images undergo the same standardized preprocessing operations as in S102 to generate standardized interface images.

[0230] The standardized interface image is input into the pre-trained YOLOv8-pose traction interface keypoint detection network, which outputs the pixel coordinates of the four corner points of the traction interface. The backbone network of the improved YOLOv8-pose network adopts the same cross-stage local network fusion structure (C2f) as S103, which includes five feature extraction stages at different scales, with output channels of 64, 128, 256, 512, and 1024, respectively; the neck network adopts a path aggregation network-feature pyramid network (PAN-FPN) structure and embeds a convolutional block attention module (CBAM); the detection head adopts a decoupled head structure, and simultaneously outputs keypoint coordinates and confidence scores. The training dataset contains 8,000 annotated traction interface images, covering interface images with different distances, angles, lighting, and wear levels. The training set accounts for 80%, the validation set for 10%, and the test set for 10%. The training batch size is preset to 16, the initial learning rate is preset to 0.01, a cosine annealing learning rate scheduling strategy is adopted, the maximum number of iterations is preset to 200 rounds, the optimizer is AdamW, and the weight decay coefficient is preset to 0.0005. After training, the model achieves a keypoint detection accuracy of 99.2% on the test set, with an average localization error of less than 3mm.

[0231] By combining the camera intrinsic parameter matrix and the pre-calibrated hand-eye calibration matrix, coordinate transformation is performed. First, the pixel coordinates of the corner points of the traction interface are converted into three-dimensional coordinates in the camera coordinate system, and then converted into three-dimensional coordinates in the traction equipment coordinate system. The coordinates of the center of the traction interface in the traction equipment coordinate system are then calculated. and the orientation angle of the traction interface The orientation angle is obtained by fitting the long side direction of the interface rectangle to the coordinates of the four corner points.

[0232] Calculate the visual deviation between the traction equipment coupling mechanism and the traction interface. Specifically, calculate the lateral deviation, longitudinal deviation, and angular difference between the centers of the coupling mechanism and the traction interface on the horizontal plane. Longitudinal deviation and angle deviation The PID controller is used to correct the engagement posture of the traction equipment in real time. The proportional gain of the PID controller is... The default value is 0.8, which is the integral coefficient. The default value is 0.1, and the differential coefficient is... The default value is 0.05.

[0233] When the lateral deviation is less than 10mm of the preset attachment threshold, the longitudinal deviation is less than 5mm of the preset attachment threshold, and the angular deviation is less than 2° of the preset attachment threshold, the traction device is controlled to stop moving, and the attachment posture adjustment is completed.

[0234] S503: Traction coupling verification and connection status confirmation.

[0235] The electric coupling mechanism of the traction equipment is controlled to perform the coupling action. The extension speed of the electric coupling mechanism is preset to 50mm / s, and the maximum extension distance is preset to 200mm. When the coupling mechanism contacts the traction interface and reaches the preset coupling force of 500N, the extension action stops.

[0236] Multiple verifications of the connection status are performed: First, the force sensor built into the hooking mechanism detects whether the hooking force is maintained within the range of 400N to 600N. This force range corresponds to the force range of the normal hooking state and can effectively identify abnormal states such as loose connection and disconnection. Second, visual inspection is used to check whether the relative position of the traction interface and the hooking mechanism remains stable within 10 consecutive frames, with a displacement change of less than 2mm. Finally, a traction test is conducted using the traction equipment, pulling the faulty AGV with a pulling force of 50N to detect whether there is a relative displacement between the two.

[0237] Once all verification items pass, the traction connection is confirmed to be successful, and the connection success flag is set. If any verification fails, the coupling mechanism is retracted, and the coupling attitude correction steps of S502 are re-executed, with a maximum of 3 retries. If all 3 retries fail, a coupling failure alarm is generated, and the current towing mission is terminated.

[0238] S504: Dynamic adjustment control of traction transfer process.

[0239] After successful traction connection, the traction equipment is controlled to move along the traction execution corridor to the traction release area at a speed of 0.3 m / s. During the transfer, the status data of the faulty AGV and the channel environment data are collected in real time at a frequency of 10 Hz, and multi-dimensional dynamic adjustments are performed.

[0240] Vehicle body yaw adjustment. The yaw angle of the faulty AGV is monitored in real time using a global industrial camera array. The difference between the current yaw angle and the tangent angle of the traction path is calculated to obtain the vehicle body yaw angle. .when At the same time, adjust the travel speed and steering angle of the traction equipment, and the speed adjustment amount. Steering angle adjustment amount The vehicle body yaw angle is controlled within ±5°.

[0241] Connection offset adjustment. The relative displacement between the coupling pin and the traction interface is detected by the displacement sensor built into the coupling mechanism to obtain the connection offset. .when If the misalignment occurs, reduce the traction equipment speed to 0.1 m / s and fine-tune the traction direction until the connection offset is less than 20 mm. If the connection offset continues to exceed 50 mm and cannot be corrected, immediately stop traction and generate a connection abnormality alarm.

[0242] Channel occupancy adjustment. Real-time calculation of the maximum occupancy width of the combination of traction equipment and faulty AGV at the current position. .when When driving, reduce the speed of the traction equipment to 0.2 m / s and keep the traction equipment traveling along the center line of the passage, ensuring that the remaining width on both sides is greater than or equal to 300 mm. Among them, 0.8 corresponds to narrow passage conditions, requiring slowing down and careful passage to avoid scratches. The remaining width of 300 mm is the minimum clearance for safe passage, which can prevent the traction assembly from scratching the boundaries of the passage on both sides.

[0243] S505: Generation of traction release and disposal results.

[0244] When the traction equipment reaches the center of the traction release area, control the traction equipment to stop moving. Execute the traction release operation: First, maintain the braking state for 3 seconds to ensure the faulty AGV is completely stationary; then control the electric coupling mechanism to retract at a speed of 50mm / s, disconnecting it from the traction interface.

[0245] Execute release status verification: Use a global industrial camera to detect whether the faulty AGV is stably stationary within the boundary of the traction release area; use a force sensor on the coupling mechanism to detect whether the tension has dropped to 0N to confirm that the connection has been completely released; use visual inspection to check whether the distance between the traction equipment and the faulty AGV is greater than or equal to 500mm to meet the safety distance requirements.

[0246] Once the release status verification is successful, a fault traction handling result is generated, including the faulty AGV's unique identifier, fault occurrence time, fault type combination, traction start time, traction end time, traction path length, any abnormal situations encountered during traction, and corresponding handling measures. The handling result is recorded in the system operation and maintenance database, and a fault handling completion notification is sent to the AGV scheduling system, updating the AGV's status to "pending maintenance."

[0247] This embodiment acquires scene images and task status of the AGV's operating area, performs visual analysis on the AGV body, channel boundaries, cargo outline, and traction interface, and aligns them temporally with the task status to form operational perception data. Based on the operational perception data, it identifies stagnation anomalies, posture deviations, and channel occupancy states. Combining continuous frame displacement changes, it identifies faulty AGVs and generates a fault pose spectrum including the vehicle center, deflection angle, interface orientation, and occupancy boundary. Based on the fault pose spectrum, it reconstructs the traction interface, calculates traction adaptation values ​​by integrating multi-dimensional traction influencing factors, and generates a traction execution corridor by combining traction constraint space and particle swarm optimization. Finally, it corrects the attachment posture based on visual deviations and dynamically adjusts the traction action during the traction transfer process to achieve accurate positioning, reliable access, safe transfer, and generation of disposal results for faulty AGVs.

[0248] Example 2

[0249] This invention discloses a vision-based AGV fault location and traction system, comprising a running perception module, a pose positioning module, an adaptation calculation module, a corridor optimization module, and a traction control module:

[0250] The operation perception module acquires scene images of the AGV's operating area and the AGV's task status. It performs visual analysis on the AGV body, channel boundaries, cargo outline, and traction interface in the scene images and aligns them with the AGV's task status in time to form operation perception data.

[0251] The pose localization module identifies AGV stagnation anomalies, posture deviations, and channel occupancy states based on operational perception data. It determines faulty AGVs by combining continuous frame displacement changes and extracts the vehicle center, deflection angle, interface orientation, and occupancy boundary of the faulty AGV to form a fault pose spectrum.

[0252] The adaptation calculation module reconstructs the traction interface of the faulty AGV based on the fault position spectrum, and integrates the traction interface orientation, vehicle body deflection angle, remaining channel width and obstacle proximity relationship to calculate the traction adaptation value.

[0253] The corridor optimization module constructs a traction constraint space based on traction adaptation values, faulty AGV occupancy boundaries, current position of traction equipment, and traction release area. It then performs particle swarm optimization using access deviation, path detour cost, corner restriction risk, and channel occupancy risk as optimization factors to generate the traction execution corridor.

[0254] The traction control module controls the traction equipment to approach the faulty AGV along the traction execution corridor, corrects the coupling posture based on the visual deviation between the traction interface and the traction equipment, and adjusts the traction action according to the vehicle body sway, connection offset and channel occupancy changes during the traction transfer process, thus forming the fault traction handling result.

[0255] refer to Figure 4 , Figure 4 This is a diagram of the architecture of a vision-based AGV fault location and traction system.

[0256] Figure 4 After the scene image and AGV task status input are displayed, they are processed sequentially by the operation perception module, pose positioning module, adaptation calculation module, corridor optimization module and traction control module to form the overall data flow relationship of operation perception data, fault pose spectrum, traction adaptation value, traction execution corridor and fault traction handling results.

[0257] It should be noted that, Figure 4 This is for illustrative purposes only, intended to help understand the process structure and data organization, and does not represent the precise working state in actual operation.

[0258] The specific functional implementation of each module is described in the relevant content of the AGV fault location and traction method based on visual recognition in Example 1, and will not be repeated here.

[0259] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A visual recognition-based AGV fault location and traction method, characterized in that, Includes the following steps: Acquire scene images of the AGV operating area and AGV task status, perform visual analysis and temporal alignment of AGV body, channel boundary, cargo outline and traction interface to form operation perception data; Based on operational perception data, stagnation anomalies, posture deviations, and channel occupancy states are identified. Combined with continuous frame displacement changes, faulty AGVs are determined and pose features are extracted to form a fault pose spectrum. The traction interface is reconstructed based on the fault position spectrum. The traction interface orientation, vehicle body deflection angle, remaining channel width and obstacle proximity relationship are fused to calculate the traction adaptation value. Based on the traction adaptation value, a traction constraint space is constructed. Particle swarm optimization is performed with access deviation, path detour cost, corner restriction risk and channel occupancy risk as optimization factors to generate a traction execution corridor. The control traction equipment completes visual correction of the coupling posture and dynamic traction transfer based on the traction execution corridor, forming a fault traction handling result.

2. The AGV fault location and traction method based on visual recognition as described in claim 1, characterized in that, The formation of the fault pose spectrum includes: Continuous time-series data sequences of AGVs are extracted from operational perception data, and the displacement of the vehicle center, changes in vehicle deflection angle, changes in traction interface orientation, channel occupancy area, and changes in task execution progress are calculated to form abnormal state time-series characteristics. Based on the timing characteristics of abnormal states, stagnation anomalies, attitude deviations, and channel occupancy states are identified, and the faulty AGV is confirmed by combining the displacement changes and task state changes of consecutive frames before and after the occurrence of the abnormal state. Extract the vehicle center, deflection angle, interface orientation, and occupied boundary of the faulty AGV. Perform accuracy verification and coordinate standardization on the extraction results to form a fault position and pose spectrum.

3. The AGV fault location and traction method based on vision recognition as described in claim 2, characterized in that, Extraction of occupied boundaries includes: The actual rectangle occupied by the vehicle body is generated based on the vehicle body center and deflection angle of the faulty AGV, and the actual rectangle occupied by the cargo is generated based on the cargo outline center and cargo deflection angle. Merge the actual rectangle occupied by the vehicle body and the actual rectangle occupied by the cargo to obtain the occupancy boundary that includes the outer contour of the vehicle body and the outer extension of the cargo. The stability of the occupied boundary is checked based on the pose changes of adjacent frames, and the checked occupied boundary is written into the fault pose spectrum.

4. The AGV fault location and traction method based on vision recognition as described in claim 1, characterized in that, The calculation of the traction adaptation value includes: The vehicle center, deflection angle, traction interface center, traction interface orientation, and occupied boundary are extracted from the fault pose spectrum, and the channel boundary and obstacle contour are extracted from the operation perception data. Establish a local coordinate system for the traction interface with the center of the traction interface and the orientation of the traction interface, transform the occupied boundary to the local coordinate system of the traction interface, and reconstruct the traction interface and traction constraint boundary; The traction interface orientation, vehicle body deflection angle, remaining channel width, and obstacle proximity are quantified separately and then fused according to the degree of traction impact to obtain the traction adaptation value.

5. The AGV fault location and traction method based on vision recognition as described in claim 4, characterized in that, The reconstruction of the traction interface includes: Based on the center and orientation of the traction interface, the local coordinate reference is determined, and the occupied boundary of the faulty AGV is mapped as a traction constraint polygon. Within the local coordinate system of the traction interface, the traction interface, the vehicle body outline, the cargo outline, and the area adjacent to obstacles are expressed, forming a correspondence between the traction direction and the space occupation; Based on the traction adaptation value, feasible traction directions are selected, and the optimal traction direction is determined to form a traction interface for constructing the traction constraint space.

6. The AGV fault location and traction method based on vision recognition as described in claim 1, characterized in that, The construction of the traction constraint space includes: Extract the optimal traction direction, traction interface center, and traction constraint boundary from the traction interface, and determine the basic constraint parameters by combining the faulty AGV occupancy boundary, the current position of the traction equipment, the traction release area, and the channel topology information; Based on the basic constraint parameters, a geometric constraint layer, an adaptive constraint layer, a kinematic constraint layer, and a safety constraint layer are constructed. Low-adaptability areas, high-risk areas, areas occupied by obstacles, and areas outside the passage boundary are uniformly marked as impassable areas, forming a traction constraint space.

7. The AGV fault location and traction method based on vision recognition as described in claim 6, characterized in that, The generation of the traction execution corridor includes: Candidate traction paths are constructed by using the sequence of path points from the current position of the traction equipment to the traction release area as particle codes. The comprehensive traction cost of the candidate traction path is calculated based on access deviation, path detour cost, corner restriction risk, and channel occupancy risk. Particle swarm optimization is performed within the traction constraint space to obtain the optimal traction path. After smoothing the optimal traction path, a traction execution corridor is generated.

8. The AGV fault location and traction method based on vision recognition as described in claim 7, characterized in that, The generation of the traction execution corridor also includes: Using the optimal traction path as the centerline, calculate the allowable offsets on both sides of the centerline relative to the channel boundary and obstacles. The left and right boundaries of the traction execution corridor are generated based on the allowed offset, and it is verified whether the traction execution corridor is located within the allowed passage area; The verified traction execution corridor is output in a standardized manner according to the boundary vertex coordinates, centerline coordinates, path length, maximum turning angle, and comprehensive traction cost.

9. The AGV fault location and traction method based on vision recognition as described in claim 1, characterized in that, Correction of the attachment posture includes: The traction equipment is controlled to approach the faulty AGV along the traction execution corridor, and an image of the traction interface area is acquired when it approaches the traction interface. Key point detection is performed on the image of the traction interface area to obtain the corner point position of the traction interface, and then the center position and orientation of the traction interface relative to the traction equipment are obtained. The attachment posture of the traction equipment is corrected based on the lateral, longitudinal, and angular deviations between the attachment mechanism and the traction interface.

10. A vision-based AGV fault location and traction system, used to implement the vision-based AGV fault location and traction method described in claims 1-9, characterized in that, It includes a motion sensing module, a pose localization module, an adaptation calculation module, a corridor optimization module, and a traction control module: The operation perception module acquires scene images of the AGV's operating area and the AGV's task status, performs visual analysis and temporal alignment of the AGV body, channel boundaries, cargo outline, and traction interface, and forms operation perception data. The pose localization module identifies stagnation anomalies, posture deviations, and channel occupancy states based on operational perception data. It combines continuous frame displacement changes to determine faulty AGVs and extracts pose features to form a fault pose spectrum. The adaptation calculation module reconstructs the traction interface based on the fault position spectrum, and integrates the traction interface orientation, vehicle body deflection angle, remaining channel width and obstacle proximity relationship to calculate the traction adaptation value. The corridor optimization module constructs a traction constraint space based on the traction adaptation value, and performs particle swarm optimization with access deviation, path detour cost, corner restriction risk and channel occupancy risk as optimization factors to generate a traction execution corridor. The traction control module controls the traction equipment to visually correct the coupling posture and dynamically traction transfer according to the traction execution corridor, thus forming a fault traction handling result.

Images