An Industrial Vision-Based Method and System for Obstacle Avoidance Scheduling of Unmanned Forklifts

Abstract

This invention discloses an obstacle avoidance scheduling method and system for unmanned forklifts based on industrial vision, belonging to the field of industrial vision technology. The method includes: acquiring continuous image data and preprocessing it to obtain an image frame sequence; constructing an improved RTMDet model to obtain target detection results; performing cross-frame association processing to output dynamic and static obstacle sets; performing look-ahead insertion and rollback verification to obtain scheduling instruction information; obtaining local control instructions based on optimal reciprocal collision avoidance; selecting the minimum cost trajectory, converting it into chassis control quantities, and issuing them for execution. This invention achieves a stable conversion of industrial vision perception results into executable obstacle avoidance trajectories and control instructions for unmanned forklifts by improving the adaptive inference structure of the RTMDet model and the vehicle-type executable decision of optimal reciprocal collision avoidance.

Description

Technical Field

[0001] This invention relates to the field of industrial vision technology, and in particular to an unmanned forklift obstacle avoidance scheduling method and system based on industrial vision. Background Technology

[0002] In existing warehousing and workshop logistics scenarios, unmanned forklifts typically rely on onboard vision or multi-sensor perception to achieve target detection, localization, and obstacle avoidance control. Due to the characteristics of the operating environment, such as rapidly changing lighting, interference from reflective materials, repetitive shelf textures, mixed traffic of personnel and vehicles, frequent occlusion, and random occurrence of temporary stacked items, industrial vision perception results are prone to target omissions, false detections, and cross-frame jitter. This leads to discontinuous obstacle state estimation, making it difficult to stably and in real-time convert perception results into executable obstacle avoidance trajectories and chassis control commands within the control cycle. When considering heavy-load conditions, minimum turning radius, braking distance, and steering response characteristics of vehicle motion, there is a gap between perception output and executable motion, often manifesting as frequent sudden stops, unstable detours, or local trajectories that cannot land smoothly, thus affecting operational efficiency and safety.

[0003] In existing unmanned forklift systems, obstacle avoidance control and fleet scheduling often employ relatively independent processing links. The scheduling layer focuses on task allocation, path distribution, and resource allocation, while the obstacle avoidance layer focuses on local collision avoidance and real-time control for individual vehicles. Due to the lack of unified modeling and collaborative decision-making regarding dynamic obstacles, traffic resource occupancy status, and multi-vehicle interaction relationships, conflicts frequently arise in narrow passages, intersections, meeting points, and one-way traffic sections, leading to multiple vehicles competing for lanes, waiting for each other, stalemates between vehicles, and conflicts between local obstacle avoidance and global paths. This further triggers congestion, deadlocks, or frequent replanning, resulting in a decrease in overall throughput and an increase in task delays. The scheduling layer typically plans based on static maps and estimated travel times, making it difficult to absorb dynamic obstacle changes perceived by visual perception in a timely manner. This causes inconsistencies between the planned traffic flow and the actual situation on site, resulting in repeated waiting, fluctuating detour paths, and delayed resource occupancy records. Local collision avoidance strategies, without considering right-of-way priority, passage order, and task urgency, are prone to non-optimal behaviors such as mutual yielding or close approaching each other, increasing collision risk and equipment wear and tear, and failing to meet the needs of high-density, multi-tasking industrial logistics operations.

[0004] Therefore, how to provide an obstacle avoidance scheduling method and system for unmanned forklifts based on industrial vision is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an obstacle avoidance scheduling method and system for unmanned forklifts based on industrial vision. This invention comprehensively utilizes industrial vision target detection, cross-frame associated state construction, time-series occupancy scheduling of passage sections, and vehicle-to-vehicle reciprocal collision avoidance decision-making technology. It details the entire process from acquiring continuous image data from an onboard industrial camera, through preprocessing, improved detection and inference, output of dynamic and static obstacle sets, conflict section identification and cost calculation scheduling, local collision avoidance decision-making, to trajectory generation and chassis control issuance. In terms of model structure, it innovatively constructs an improved RTMDet model and introduces segmented early-retreat paths, conditional computational routing, and main-neck collaborative rearrangement. In terms of collision avoidance structure, it constructs optimal reciprocal collision avoidance and includes variable responsibility allocation components and vehicle motion reachability domain solutions, achieving stable real-time conversion of perception results into executable control commands and conflict suppression and congestion reduction under multi-vehicle parallel operation conditions. Compared with existing technologies, this invention has advantages such as high real-time performance, stable connection between perception and control, improved fleet passage efficiency, and strong adaptability to complex dynamic industrial environments.

[0006] An obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to an embodiment of the present invention includes:

[0007] Collect continuous image data from the industrial camera mounted on the unmanned forklift, preprocess the continuous image data to obtain an image frame sequence;

[0008] An improved RTMDet model is constructed to perform target detection on image frame sequences. The segmented inference path selection is performed based on the segmented early retreat path to determine the inference depth. The scenario adaptive routing is performed using conditional computing power routing to determine the computation sub-path. The collaborative rearrangement is performed based on the main neck collaborative rearrangement to complete feature generation and fusion, and the target detection results are obtained.

[0009] Cross-frame association processing is performed on the target detection results. A cross-frame candidate association set is constructed in the uniform coordinate system where the vehicle is stationary, and an association cost tensor is generated. Velocity-acceleration continuity correction is performed based on the association cost tensor, and dynamic and static obstacle sets are output.

[0010] The availability status of the access sections in the work area is updated by dynamic and static obstacle sets. Look-ahead insertion and rollback verification are performed to identify conflict sections. The waiting cost and detour cost of the conflict sections are calculated to obtain scheduling instruction information.

[0011] Based on optimal reciprocal collision avoidance, local collision avoidance decision processing is performed on dynamic and static obstacle sets and scheduling instruction information. The responsibility ratio is determined by a variable responsibility allocation component. The feasible action is solved by solving the reachability domain of vehicle motion and the optimal action is selected to obtain local control instructions.

[0012] A discrete control sequence is generated based on the local control command, a forward time-domain rolling simulation is performed, the minimum cost trajectory is selected, the minimum cost trajectory is converted into a chassis control quantity and issued for execution.

[0013] Optionally, the continuous image data includes timestamp sequences, image frame pixel matrices, camera intrinsic data, camera extrinsic data, lens distortion parameter data, and camera exposure and gain parameter data.

[0014] Optionally, obtaining the image frame sequence includes:

[0015] The system collects continuous image data output from the industrial camera on the unmanned forklift. The continuous image data includes timestamp sequence, image frame pixel matrix, camera intrinsic parameter data, camera extrinsic parameter data, lens distortion parameter data, and camera exposure and gain parameter data. The continuous image data is time-aligned to form an image frame sequence sorted by timestamp.

[0016] Distortion correction is performed on the image frame pixel matrix based on lens distortion parameter data, and brightness normalization is performed on the image frame pixel matrix based on camera exposure and gain parameter data. The distortion-corrected image frame pixel matrix and the brightness-normalized image frame pixel matrix are then unified into the target detection input format.

[0017] Based on camera intrinsic and extrinsic data, coordinate parameter binding processing is performed on the image frame sequence to generate image frames associated with calibration information.

[0018] Optionally, obtaining the target detection result includes:

[0019] An improved RTMDet model is constructed. The improved RTMDet model sets up a segmented early retreat path and a main-neck collaborative rearrangement structure between the backbone feature extraction structure and the neck feature fusion structure of the original RTMDet model. Conditional computing power routing is set up inside the original backbone feature extraction structure.

[0020] The image frame sequence is processed by segmented early termination path to obtain the intra-frame target density index, intra-frame texture complexity index and inter-frame brightness change index of the image frame sequence, forming an inference depth discriminant. Based on the inference depth discriminant, the target early termination point is determined in the early termination point of the segmented early termination path. The target detection inference calculation is completed within the inference depth range corresponding to the target early termination point to obtain the target detection result.

[0021] Conditional computing power routing processing is performed on the feature extraction process within the inference depth range corresponding to the early exit of the target. A scene routing identifier is generated based on the intermediate features corresponding to the early exit of the target. The target computing sub-path is determined in the parallel computing sub-path based on the scene routing identifier. Feature update and transmission are completed within the target computing sub-path to obtain routing features.

[0022] The routing features are processed by main-neck collaborative rearrangement. A computational dependency table is established between each output node of the main feature extraction structure and each fusion node of the neck feature fusion structure. The computational execution order of the main feature extraction structure and the neck feature fusion structure is determined according to the computational dependency table, and the computational order is rearranged. Feature generation and feature fusion are completed according to the computational execution order to obtain fused features. The target detection results are output based on the fused features.

[0023] The improved RTMDet model is trained by constructing a training sample set and batch inputting the training sample set, calculating the detection loss based on the bounding boxes and category labels of the training sample set, performing backpropagation to update the model parameters, and completing the training of the improved RTMDet model.

[0024] Optionally, the output set of dynamic and static obstacles includes:

[0025] Based on the unmanned forklift wheel speed, rudder angle and timestamp sequence, the planar pose increment of the unmanned forklift between two adjacent frames is calculated. The center point of the bounding box in the target detection result is transformed according to the planar pose increment to obtain the set of detection points in the unified coordinate system of the stationary vehicle body.

[0026] In a stationary, unified coordinate system of the vehicle body, a cross-frame candidate association set is constructed using the set of detection points as nodes, and an association cost tensor is generated for each candidate pairing relationship.

[0027] Based on the association cost tensor, cross-frame global association is performed on the cross-frame candidate association set to obtain the target trajectory set. Based on the target trajectory set, the velocity sequence and acceleration sequence of each target trajectory at adjacent time points are calculated. Continuity correction is performed on the velocity sequence and acceleration sequence to obtain the target state information. Based on the target state information, the target trajectory set is divided into dynamic and static obstacle sets.

[0028] Optionally, obtaining the scheduling instruction information includes:

[0029] A set of access sections is generated based on the work area map, and the availability status of each access section is determined.

[0030] Receive the task queue and the current status of the unmanned forklift and generate an initial passage plan. The initial passage plan includes the sequence of passage sections and the corresponding time windows. Write the initial passage plan into the time-series occupancy table. The time-series occupancy table records the time windows occupied by the unmanned forklift in each passage section, with the passage section as the index.

[0031] The initial passage plan is inserted forward and conflict sections are identified. The time window to be inserted is compared with the existing time window of the same passage section in the time-series occupancy table. The passage section that meets the judgment condition is identified as a conflict section. The conflict section is rolled back and verified to generate a set of alternative passage plan candidates.

[0032] Calculate the waiting cost and detour cost of the conflict section, select the alternative passage plan with the minimum sum of waiting cost and detour cost from the alternative passage plan candidate set as the scheduling result, and output the scheduling instruction information including the passage section sequence, the occupied time window and the waiting or detour instructions.

[0033] Optionally, obtaining the local control instructions includes:

[0034] The system receives information on the unmanned forklift's own status, dynamic obstacle set, and scheduling instructions. The unmanned forklift's own status includes its current position, current heading, current linear velocity, and current turning state. Each dynamic obstacle in the dynamic obstacle set includes its position, velocity, and equivalent occupied radius.

[0035] A variable responsibility allocation component is used to construct an interaction pair between the unmanned forklift and each dynamic obstacle. A responsibility ratio parameter is calculated for each interaction pair. The responsibility ratio parameter is used to determine the avoidance share of the unmanned forklift in the interaction pair.

[0036] Based on the relative position and relative speed of the unmanned forklift and dynamic obstacles, a collision speed prohibition region is constructed within the future prediction time window. The collision speed prohibition region is transformed into an allowed speed half-plane in the speed plane. The intersection of all interaction pairs with the corresponding allowed speed half-planes is obtained to obtain the joint allowed speed region.

[0037] By solving the reachability domain of vehicle motion, local control commands are selected within the joint permissible speed domain. The optimal action is then solved within the intersection of the reachability domain of vehicle motion and the joint permissible speed domain. The criterion for solving the optimal action is to minimize the sum of the square of the difference between the linear velocity and the reference velocity and the square of the difference between the angular velocity and the reference angular velocity. The linear velocity and angular velocity corresponding to the optimal action are then output as local control commands.

[0038] Optionally, the step of selecting the minimum cost trajectory, converting the minimum cost trajectory into a chassis control quantity, and issuing it for execution includes:

[0039] It receives local control commands and generates discrete control sequences. The discrete control sequence consists of multiple control steps. Each control step includes linear velocity commands and angular velocity commands. The time interval between control steps is a fixed control period.

[0040] Forward time-domain rolling simulation is performed on the discrete control sequence to obtain short-time trajectory clusters. The forward time-domain rolling simulation iteratively calculates the trajectory point sequence for each control step according to the vehicle kinematics update rule. The vehicle kinematics update rule is that the x-coordinate of the current position is equal to the x-coordinate of the previous moment plus the linear velocity multiplied by the cosine of the heading angle and then multiplied by the control period; the y-coordinate of the current position is equal to the y-coordinate of the previous moment plus the linear velocity multiplied by the sine of the heading angle and then multiplied by the control period; and the heading angle is equal to the heading angle of the previous moment plus the angular velocity and then multiplied by the control period.

[0041] The cost is calculated for short-time trajectory clusters. The cost consists of trajectory smoothing cost, braking margin cost, and boundary fitting cost. The trajectory point sequence with the minimum cost is selected as the minimum cost trajectory.

[0042] The minimum cost trajectory is converted into chassis control quantities and issued for execution. The chassis control quantities include linear speed control quantities and steering control quantities. Execution feedback information is collected, which includes actual linear speed, actual steering status, and actual trajectory point sequence.

[0043] An unmanned forklift obstacle avoidance scheduling system based on industrial vision according to an embodiment of the present invention includes the following modules:

[0044] The image acquisition and preprocessing module is used to acquire continuous image data from the vehicle-mounted industrial camera and output an image frame sequence after preprocessing.

[0045] An improved detection inference module is used to perform target detection on image frame sequences based on an improved RTMDet model and output the target detection results.

[0046] The cross-frame correlation modeling module is used to perform cross-frame correlation and velocity-acceleration continuity correction on target detection results, and output dynamic and static obstacle sets;

[0047] The traffic scheduling generation module is used to update the availability status of the passage section based on dynamic and static obstacle sets and output scheduling instruction information.

[0048] The collision avoidance decision module is used to perform local collision avoidance decisions based on the optimal reciprocal collision avoidance set of dynamic and static obstacles and scheduling instruction information, and output local control instructions.

[0049] The trajectory generation and execution module is used to generate discrete control sequences based on local control commands, perform forward time-domain rolling simulation to select the minimum cost trajectory, and then issue chassis control quantities for execution.

[0050] The beneficial effects of this invention are:

[0051] This invention improves the continuity and stability of obstacle perception in dynamic industrial scenarios by preprocessing continuous image data from industrial cameras, improving target detection using the RTMDet model, and constructing cross-frame associated states. Compared with existing schemes that rely solely on single-frame detection or have a disconnect between perception and control, this invention achieves adaptive selection of inference depth based on segmented early-retreat paths, adaptive switching of computational paths under different operating conditions based on conditional computing power routing, and improves feature generation and fusion efficiency based on main-neck collaborative rearrangement. This maintains high real-time performance and detection stability in scenarios with varying illumination, glare interference, and frequent occlusion. Simultaneously, by constructing and performing velocity-acceleration continuity correction through cross-frame candidate association and association cost tensor, the obstacle state output becomes more coherent, reducing the impact of false detections, missed detections, and jitter on subsequent obstacle avoidance control, and achieving stable input of perception results to executable motion decisions.

[0052] This invention achieves traffic order coordination and congestion suppression during multi-vehicle parallel operations by updating the availability status of traffic sections, identifying conflict sections through look-ahead insertion and rollback verification, and generating scheduling instructions based on waiting costs and detour costs. Compared with existing scheduling schemes that only output global paths and whose independent local obstacle avoidance decisions lead to meeting conflicts and deadlocks, this invention incorporates dynamic and static obstacle sets into the traffic resource availability update and uses a time-series occupancy table to plan and verify the occupancy of traffic sections, forming executable scheduling instructions. Furthermore, by combining the variable responsibility allocation component in optimal reciprocal collision avoidance with the solution of vehicle motion reachability domain, local collision avoidance decisions can conform to right-of-way and traffic order, and output control instructions that can be directly executed by the vehicle chassis. This reduces oncoming standoffs, frequent starts and stops, and repeated detours at narrow passages and intersections, improving overall throughput efficiency and operational safety. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a flowchart of an unmanned forklift obstacle avoidance scheduling method based on industrial vision proposed in this invention;

[0055] Figure 2 This is a structural block diagram of the improved RTMDet model of an unmanned forklift obstacle avoidance scheduling method based on industrial vision proposed in this invention.

[0056] Figure 3 This is a functional schematic diagram of an unmanned forklift obstacle avoidance scheduling system based on industrial vision proposed in this invention. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0058] refer to Figure 1 and Figure 2 An obstacle avoidance scheduling method for unmanned forklifts based on industrial vision includes:

[0059] Collect continuous image data from the industrial camera mounted on the unmanned forklift, preprocess the continuous image data to obtain an image frame sequence;

[0060] An improved RTMDet model is constructed to perform target detection on image frame sequences. The segmented inference path selection is performed based on the segmented early retreat path to determine the inference depth. The scenario adaptive routing is performed using conditional computing power routing to determine the computation sub-path. The collaborative rearrangement is performed based on the main neck collaborative rearrangement to complete feature generation and fusion, and the target detection results are obtained.

[0061] Cross-frame association processing is performed on the target detection results. A cross-frame candidate association set is constructed in the uniform coordinate system where the vehicle is stationary, and an association cost tensor is generated. Velocity-acceleration continuity correction is performed based on the association cost tensor, and dynamic and static obstacle sets are output.

[0062] The availability status of the access sections in the work area is updated by dynamic and static obstacle sets. Look-ahead insertion and rollback verification are performed to identify conflict sections. The waiting cost and detour cost of the conflict sections are calculated to obtain scheduling instruction information.

[0063] Based on optimal reciprocal collision avoidance, local collision avoidance decision processing is performed on dynamic and static obstacle sets and scheduling instruction information. The responsibility ratio is determined by a variable responsibility allocation component. The feasible action is solved by solving the reachability domain of vehicle motion and the optimal action is selected to obtain local control instructions.

[0064] A discrete control sequence is generated based on the local control command, a forward time-domain rolling simulation is performed, the minimum cost trajectory is selected, the minimum cost trajectory is converted into a chassis control quantity and issued for execution.

[0065] In this embodiment, the continuous image data includes timestamp sequences, image frame pixel matrices, camera intrinsic data, camera extrinsic data, lens distortion parameter data, and camera exposure and gain parameter data.

[0066] In this embodiment, obtaining the image frame sequence includes:

[0067] The system collects continuous image data output from the industrial camera on the unmanned forklift. The continuous image data includes timestamp sequence, image frame pixel matrix, camera intrinsic parameter data, camera extrinsic parameter data, lens distortion parameter data, and camera exposure and gain parameter data. The continuous image data is time-aligned to form an image frame sequence sorted by timestamp.

[0068] Distortion correction is performed on the image frame pixel matrix based on lens distortion parameter data, and brightness normalization is performed on the image frame pixel matrix based on camera exposure and gain parameter data. The distortion-corrected and brightness-normalized image frame pixel matrices are then unified into the target detection input format. Specifically, the brightness normalization process based on camera exposure and gain parameter data is as follows:

[0069] The camera reads the exposure time, analog gain, and digital gain of the current frame from the camera metadata, multiplies them to obtain the effective exposure coefficient of the current frame, and performs exposure compensation for each pixel value in the image frame pixel matrix by dividing the original pixel value by the effective exposure coefficient to obtain the compensated pixel matrix. The brightness mean and brightness standard deviation of the whole frame are calculated on the compensated pixel matrix. The brightness mean is the arithmetic mean of all pixel values, and the brightness standard deviation is the square root of the average of the squares of the differences between each pixel value and the brightness mean. Then, the normalization process is performed on each pixel by subtracting the brightness mean from the compensated pixel value and then dividing by the brightness standard deviation to obtain the normalized pixel matrix. The normalized pixel matrix is ​​linearly mapped to the range of 0 to 255, and values ​​less than 0 are truncated to 0, and values ​​greater than 255 are truncated to 255. The brightness-normalized image frame pixel matrix is ​​then output.

[0070] Based on camera intrinsic and extrinsic data, coordinate parameter binding processing is performed on the image frame sequence to generate image frames associated with calibration information.

[0071] In this embodiment, obtaining the target detection result includes:

[0072] An improved RTMDet model is constructed. The improved RTMDet model sets up a segmented early retreat path and a main-neck collaborative rearrangement structure between the backbone feature extraction structure and the neck feature fusion structure of the original RTMDet model. Conditional computing power routing is set up inside the original backbone feature extraction structure.

[0073] A segmented early termination path processing is performed on the image frame sequence to obtain intra-frame target density, intra-frame texture complexity, and inter-frame brightness change indices, forming an inference depth discriminant. Based on the inference depth discriminant, the target early termination point is determined in the early termination point of the segmented early termination path. Target detection inference calculations are then performed within the inference depth range corresponding to the target early termination point to obtain the target detection result.

[0074] To obtain the intra-frame target density index, intra-frame texture complexity index, and inter-frame brightness change index of the image frame sequence, an inference depth discriminant metric is formed. Specifically, a fast coarse detection is performed on the current frame to obtain a set of candidate boxes. The coarse detection uses the output of the shallowest layer detector head of the improved RTMDet model. Candidate boxes with a confidence level greater than or equal to 0.3 are retained. The number of retained candidate boxes is counted and divided by the image area to obtain the number of candidates per unit area as the target density index. The intra-frame texture complexity index is obtained by converting the current frame to a grayscale image and calculating the relationship between each pixel and its right neighbor. The average gradient magnitude is obtained by summing the absolute values ​​of the grayscale differences with the next neighboring pixel and averaging them over the entire image. At the same time, the discrete entropy of the grayscale histogram is calculated as a supplementary component of the texture complexity. The texture complexity index is obtained by linearly combining the average gradient magnitude and the histogram entropy. The inter-frame brightness change index is obtained by calculating the grayscale mean and grayscale standard deviation of two adjacent frames respectively, taking the absolute difference between the grayscale mean and the absolute difference between the grayscale standard deviation of the two frames and adding them together. The three indices are concatenated in order and normalized to the range of 0 to 1 to form the inference depth discriminant.

[0075] Based on the inference depth discrimination metric, the target early exit point is determined from the early exit points of the segmented early exit path. Specifically, three early exit points are preset to correspond to shallow, medium and deep inference depths, respectively. Threshold combination rules are set for the three indicators. When the target density index is less than 0.002, the texture complexity index is less than 0.2 and the brightness change index is less than 8, the shallow early exit point is selected. When the target density index is less than 0.006, the texture complexity index is less than 0.35 and the brightness change index is less than 18, the medium early exit point is selected. In the remaining cases, the deep early exit point is selected. If the target density index is greater than or equal to 0.001 or the brightness change index is greater than or equal to 30, the deep early exit point is directly selected.

[0076] The target detection inference calculation is completed within the inference depth range corresponding to the target early exit point to obtain the target detection result. Specifically, the main feature extraction is performed until the stage output feature corresponding to the target early exit point is reached. Then, the neck feature fusion path matching the current stage is executed and connected to the corresponding detection head. When the shallow early exit point is selected, only the feature extraction of the shallow stage and one feature fusion are calculated and the detection box is output. When the middle early exit point is selected, the calculation continues to the middle stage and the middle layer feature fusion is supplemented before the detection box is output. When the deep early exit point is selected, the complete main stage and the complete neck are fused and the detection box is output. In all three cases, the confidence screening and overlapping box suppression are performed on the detection head output. The confidence screening threshold is 0.25 and the overlapping box suppression overlap threshold is 0.6. Finally, the target detection result is obtained.

[0077] Conditional computational power routing is performed on the feature extraction process within the inference depth range corresponding to the target early exit point. A scene routing identifier is generated based on the intermediate features corresponding to the target early exit point. The target computation sub-path is determined in the parallel computation sub-path based on the scene routing identifier. Feature updating and propagation are completed within the target computation sub-path to obtain routing features, where:

[0078] The scene routing identifier is generated based on the intermediate features corresponding to the target early exit. Specifically, the spatial mean and spatial standard deviation are calculated for each channel of the intermediate feature to form a channel mean vector and a channel standard deviation vector. Then, two types of statistics are calculated. The first type is the difference between the maximum and minimum values ​​of the channel mean vector, which is used to characterize the overall brightness and contrast distribution. The second type is the mean of the channel standard deviation vector, which is used to characterize the texture undulation and edge density. At the same time, the intermediate feature is divided into four quadrants according to space. The mean of the channel mean vector in each quadrant is calculated. The difference between the maximum and minimum values ​​of the four quadrant means is taken as the spatial non-uniformity, which is used to characterize the regional differences caused by occlusion and local strong reflection. The overall brightness and contrast distribution, texture undulation, and spatial non-uniformity are concatenated in sequence to obtain the scene routing identifier.

[0079] Based on the scene routing identifier, the target computation sub-path is determined in the parallel computation sub-path. Feature updates and propagation are completed within the target computation sub-path to obtain the routing features. Specifically, two parallel computation sub-paths are preset, corresponding to the simple scene path and the complex scene path, respectively. The spatial non-uniformity and texture undulation in the scene routing identifier are used as the discrimination criteria. When the spatial non-uniformity is less than 0.3 and the texture undulation is less than 0.25, the simple scene path is selected; otherwise, the complex scene path is selected. After the sub-path is determined, feature updates and propagation are performed only on the intermediate features within the target computation sub-path. The output features are calculated layer by layer according to the sub-path hierarchy and the output features are used as the input of the next layer until the exit of the inference depth range is reached to obtain the routing features.

[0080] The routing features undergo a main-neck collaborative rearrangement process. A computational dependency table is established between each output node of the main feature extraction structure and each fusion node of the neck feature fusion structure. Based on the computational dependency table, the computation execution order of the main feature extraction structure and the neck feature fusion structure is determined, and the computation order is rearranged. Feature generation and feature fusion are completed according to the computation execution order to obtain fused features. The target detection result is then output based on the fused features.

[0081] A computational dependency table is established between each output node of the main feature extraction structure and each fusion node of the neck feature fusion structure, specifically as follows:

[0082] The output nodes of the main feature extraction structure are marked according to the inference depth range. The output nodes include shallow output nodes, middle output nodes and deep output nodes. The output feature size and number of channels of each output node are recorded. The fusion nodes of the neck feature fusion structure are marked. The fusion nodes include top-down fusion nodes and bottom-up fusion nodes. The input source and output destination of each fusion node are recorded. The input port of each fusion node is traversed one by one. If the input port comes from the output node or from the output of the previous fusion node, a dependency relationship is registered in the dependency relationship table.

[0083] The dependency field includes dependency source identifier, dependency target identifier, dependency type and data form. The dependency type is used to identify two types of relationships: output to fusion and fusion to fusion. The data form is used to identify feature size and channel consistency requirements. For input ports that require scale transformation, the scale transformation operator identifier and target size corresponding to the port are added to the dependency table to form a full dependency table from the main output node to the neck fusion node.

[0084] The computation order is rearranged, and feature generation and fusion are performed according to the computation order to obtain fused features. The target detection result is then output based on the fused features. Specifically:

[0085] Based on the dependency table, a directed acyclic dependency graph is constructed for all output nodes and fusion nodes. Nodes with an in-degree of 0 are used as the set of executable nodes. An execution sequence is generated using the earliest available scheduling rule. Each time, the node whose output will be consumed by the most downstream nodes is selected from the set of executable nodes for priority execution. During the generation of the execution sequence, when the output node is calculated and the corresponding dependency is satisfied, the in-degree of the dependent target node is decremented by 1. The dependent target node with an in-degree of 0 is added to the set of executable nodes. The main output nodes are calculated in sequence according to the execution sequence and the output is cached in the shared feature buffer. The calculation of the corresponding fusion node is triggered according to the execution sequence, so that the fusion node performs fusion and outputs the fusion feature immediately after all inputs are received.

[0086] When the execution sequence reaches the fusion level corresponding to the early exit point of the target, the fusion features of the level are taken as input to the detection head to generate category and bounding box predictions. Confidence filtering and overlapping box suppression are performed on the prediction results to obtain the target detection results.

[0087] The improved RTMDet model is trained by constructing a training sample set and batch inputting the training sample set. The detection loss is calculated based on the bounding boxes and class labels of the training sample set. Backpropagation is then performed to update the model parameters, completing the training of the improved RTMDet model. Specifically, the calculation of the detection loss based on the bounding boxes and class labels of the training sample set is as follows:

[0088] For each batch of samples, a candidate prediction set is first output. The candidate predictions include class probabilities and bounding boxes. The candidate predictions are matched with the labeled boxes according to their overlap. For each labeled box, the top 10 candidates with the largest overlap of the same class are selected as positive samples, and the rest are negative samples. The class loss is calculated by cross-entropy for positive and negative samples and averaged over all samples. The regression loss is calculated only for positive samples. First, the predicted box and the labeled box are converted into center point, width and height. The sum of the absolute values ​​of the center point deviation and the width and height deviation is calculated as the regression error. Then, 1 is subtracted from the overlap as the overlap penalty. The regression error and the overlap penalty are added together and averaged over the positive samples to obtain the regression loss. The class loss and the regression loss are added together to obtain the detection loss.

[0089] In this embodiment, the output set of dynamic and static obstacles includes:

[0090] Based on the unmanned forklift's wheel speed, rudder angle, and timestamp sequence, the planar pose increment of the forklift between two adjacent frames is calculated. The center points of the bounding boxes in the target detection results are then transformed according to the planar pose increment to obtain a set of detection points in a static, unified coordinate system of the vehicle body.

[0091] The planar pose increment of an unmanned forklift between two adjacent frames is calculated based on the wheel speed, rudder angle, and timestamp sequence. Specifically, the time difference between two adjacent frames is taken as the time interval. The wheel speed sequence is averaged within the time interval to obtain the average vehicle speed. The average vehicle speed is multiplied by the time interval to obtain the travel distance within the interval. The average rudder angle sequence within the time interval is taken and combined with the vehicle wheelbase to calculate the curvature. The travel distance is multiplied by the curvature to obtain the heading change. When the absolute value of the rudder angle is less than 0.5 degrees, the curvature is considered to be 0 and the heading change is set to 0. The cosine of the travel distance multiplied by the median heading is used as the lateral displacement increment, and the sine of the travel distance multiplied by the median heading is used as the longitudinal displacement increment. The median heading is the heading of the previous frame plus half of the heading change. Finally, the planar pose increment composed of the lateral displacement increment, the longitudinal displacement increment, and the heading change is obtained.

[0092] The bounding box center points in the target detection results are transformed by the planar pose increment to obtain the detection point set in the vehicle static unified coordinate system. Specifically, each bounding box center point is converted from pixel coordinates to vehicle planar coordinates. The pixel center points are back-projected onto the ground plane according to the camera intrinsic parameters. The planar coordinate points in the vehicle coordinate system are obtained by combining the camera extrinsic parameters and the installation height. The planar coordinate points are back-compensated to the reference position of the previous frame by the planar pose increment. The planar coordinate points are rotated in the opposite direction to the heading change. The lateral displacement increment and the longitudinal displacement increment are then subtracted to obtain the coordinate points in the vehicle static unified coordinate system. The transformation is repeated for all bounding box center points in the current frame to form the detection point set in the vehicle static unified coordinate system.

[0093] In a stationary, unified coordinate system for the vehicle body, a cross-frame candidate association set is constructed using the set of detection points as nodes. For each candidate pairing, an association cost tensor is generated, where:

[0094] Constructing a cross-frame candidate association set involves taking the current frame detection point set and the historical frame detection point set in a unified coordinate system where the vehicle is stationary. Each detection point has a category identifier and timestamp. For each detection point in the current frame, select detection points of the same category from the historical frames as candidate sets. Neighborhood filtering is performed using Euclidean distance. Historical frame detection points with a distance of less than or equal to 2.0 meters are retained as candidate matching points. When the number of candidate matching points is greater than 20, only the 20 with the smallest distance are retained. The current frame detection points and candidate matching points are paired to form a candidate pairing relationship set. A corresponding candidate pairing relationship list is formed for each current frame detection point. All lists are summarized to obtain the cross-frame candidate association set.

[0095] The association cost tensor is generated as follows: For each candidate pairing relationship, three types of cost values ​​are calculated and arranged into a cost vector in a fixed order. The first type is geometric consistency cost, which is the overlap between the current frame bounding box and the historical frame bounding box, and the overlap is subtracted from 1 as the cost. The second type is motion consistency cost, which is the velocity vector calculated based on the displacement and timestamp difference of the detection points in the two frames. The cosine of the angle between the velocity direction of the current frame and the velocity direction of the historical frame is calculated, and the truncated value of the cosine is subtracted from 1 as the cost. The third type is appearance consistency cost, which is the appearance feature vector of the target region in the current frame and the historical frame is extracted and the normalized inner product is calculated. The truncated value of the normalized inner product is subtracted from 1 as the cost. The three types of cost values ​​are stacked in the order of geometric consistency cost, motion consistency cost, and appearance consistency cost, and organized into a three-dimensional array according to the detection point index of the current frame and the detection point index of the historical frame to obtain the association cost tensor.

[0096] Based on the association cost tensor, cross-frame global association is performed on the cross-frame candidate association set to obtain the target trajectory set. Based on the target trajectory set, the velocity sequence and acceleration sequence of each target trajectory at adjacent time points are calculated. Continuity correction is performed on the velocity and acceleration sequences to obtain target state information. Based on the target state information, the target trajectory set is divided into dynamic and static obstacle sets, where:

[0097] The target trajectory set is obtained by performing cross-frame global association on the cross-frame candidate association set based on the association cost tensor. Specifically, for each candidate pairing relationship, the geometric consistency cost, motion consistency cost, and appearance consistency cost are weighted and summed to form the total association cost. The weight of the geometric consistency cost is 0.4, the weight of the motion consistency cost is 0.4, and the weight of the appearance consistency cost is 0.2. A cost matrix is ​​constructed with the current frame detection point as the row and the historical frame detection point as the column. The positions in the cost matrix where there is no candidate pairing relationship are filled with 1.0. One-to-one minimum cost matching is performed on the cost matrix to obtain the set of matching pairs between the current frame detection point and the historical frame detection point. Matching pairs with a total association cost greater than 0.6 are removed. For the remaining matching pairs, the current frame detection point is appended to the end of the corresponding historical trajectory to form an updated trajectory. New trajectories are created for the current frame detection points that are not matched. The historical trajectories that are not matched are marked as missing and the missing frame count is incremented by 1. When the missing frame count is greater than 5, the trajectory is terminated, and the target trajectory set is obtained.

[0098] Based on the target state information, the target trajectory set is divided into dynamic and static obstacle sets. Specifically, for each target trajectory, the position difference and time difference between adjacent moments are calculated in a uniform coordinate system where the vehicle is stationary to obtain a velocity sequence. The velocity difference and time difference between adjacent moments are calculated to obtain an acceleration sequence. When performing continuity correction on the velocity and acceleration sequences, points in the velocity sequence where the absolute value of the velocity difference from the previous moment is greater than 1.5 m / s are replaced with linear interpolation of the velocities of the two points. Points in the acceleration sequence where the absolute value is greater than 3.0 m / s² are truncated to a range of ±3.0 m / s². After correction, the average trajectory velocity and trajectory displacement amplitude are calculated. The average trajectory velocity is the average of the velocity magnitudes at each moment in the velocity sequence, and the trajectory displacement amplitude is the Euclidean distance between the beginning and end positions of the trajectory. When the average trajectory velocity is less than 0.1 m / s and the trajectory displacement amplitude is less than 0.30 m, the target corresponding to the trajectory is assigned to the static obstacle set, and the targets corresponding to the remaining trajectories are assigned to the dynamic obstacle set.

[0099] In this embodiment, obtaining the scheduling instruction information includes:

[0100] A set of access sections is generated based on the work area map, and the availability status of each access section is determined.

[0101] Receive the task queue and the current status of the unmanned forklift and generate an initial passage plan. The initial passage plan includes the sequence of passage sections and the corresponding time windows. Write the initial passage plan into the time-series occupancy table. The time-series occupancy table records the time windows occupied by the unmanned forklift in each passage section, with the passage section as the index.

[0102] The initial traffic plan undergoes look-ahead insertion and conflict segment identification. The time window to be inserted is compared with the existing time windows of the same traffic segment in the time-series occupancy table for time window overlap detection. Traffic segments meeting the criteria are identified as conflict segments. A rollback check is performed on the conflict segments to generate a candidate set of alternative traffic plans, including time window overlap detection.

[0103] To insert a time window, first read the start and end times, then retrieve all existing time windows in the same passage segment from the time occupancy table. For each existing time window, read the start and end times, and calculate the overlap length between them. The overlap length is calculated as the time difference between the larger of the start time to be inserted and the existing start time and the smaller of the end time to be inserted and the existing end time. If the overlap length is greater than 0, it is determined that there is an overlap. If there is an overlap, the overlap ratio is further calculated. The overlap ratio is equal to the overlap length divided by the length of the time window to be inserted. If the overlap ratio is greater than or equal to 0.05, the existing time window is determined to be in conflict. As long as any existing time window in the same passage segment is determined to be in conflict, the passage segment is determined to be a conflict segment.

[0104] Calculate the waiting cost and detour cost of the conflict section. Select the alternative passage plan with the minimum sum of waiting cost and detour cost from the candidate alternative passage plan set as the scheduling result. Output scheduling instruction information including the passage section sequence, occupancy time window, and waiting or detour instructions. Specifically, the calculation of the waiting cost and detour cost of the conflict section is as follows:

[0105] For each candidate in the alternative passage plan candidate set, extract the planned entry time and the actual available entry time at the entrance of the conflict section. The actual available entry time is the maximum value of the end time of all conflict-occupied time windows in the conflict section plus a 0.5-second safety interval. The waiting time is equal to the actual available entry time minus the planned entry time. If it is less than 0, it is counted as 0. The waiting cost is equal to the waiting time multiplied by the unit time cost coefficient. The unit time cost coefficient is 1.0.

[0106] For detour candidates, the total path length of the alternative passage segment sequence is calculated. The path length is the sum of the distances of the connecting edges of each adjacent passage segment. The total path length of the original passage segment sequence is also calculated. The detour distance increment is equal to the alternative total path length minus the original total path length, and is counted as 0 if it is less than 0. The detour cost is equal to the detour distance increment multiplied by the unit distance cost coefficient, which is 0.2. The total cost is calculated for each candidate. The total cost is equal to the sum of the waiting cost and the detour cost. The candidate with the minimum total cost is selected as the scheduling result, and the corresponding waiting or detour instruction is generated.

[0107] In this embodiment, obtaining the local control command includes:

[0108] The system receives information on the unmanned forklift's own status, dynamic obstacle set, and scheduling instructions. The unmanned forklift's own status includes its current position, current heading, current linear velocity, and current turning state. Each dynamic obstacle in the dynamic obstacle set includes its position, velocity, and equivalent occupied radius.

[0109] A variable responsibility allocation component is used to construct interaction pairs between the unmanned forklift and each dynamic obstacle. A responsibility ratio parameter is calculated for each interaction pair, which determines the unmanned forklift's avoidance share within the interaction pair. Specifically, the construction of interaction pairs between the unmanned forklift and each dynamic obstacle, and the calculation of the responsibility ratio parameter for each interaction pair, are as follows:

[0110] Using the unmanned forklift as the main body, the system traverses the set of dynamic obstacles. For each dynamic obstacle, the position and velocity in the vehicle coordinate system are read, and the relative position vector, relative velocity vector, and relative distance with the unmanned forklift are calculated. When the relative distance is less than or equal to 12.0 meters and the projection of the relative velocity in the relative position direction is less than 0, and the two show a tendency to approach each other, the unmanned forklift-dynamic obstacle is registered as an interaction pair. The passage section type and right-of-way priority in the scheduling instruction of the interaction pair are recorded.

[0111] When calculating the responsibility ratio parameter, a baseline responsibility ratio of 0.5 is first assigned. Adjustments are made based on right-of-way priority: the responsibility ratio is 0.3 when the unmanned forklift has right-of-way and 0.7 when it does not. Further adjustments are made based on the type of traffic section: in narrow passages or meeting points, the responsibility ratio is reduced by 0.1 from the original value to 0.5; in intersections, it is reduced by 0.05 from the original value to 0.5. Finally, adjustments are made based on relative orientation: when a dynamic obstacle is within ±30 degrees in front of the unmanned forklift, the responsibility ratio is increased by 0.05; when it is within 30 to 120 degrees to the side, it remains unchanged; and when it is within 120 to 180 degrees behind, the responsibility ratio is decreased by 0.05. These are the final responsibility ratio parameters.

[0112] Based on the relative position and relative velocity between the unmanned forklift and dynamic obstacles, a collision velocity prohibition region is constructed within the future prediction time window. This prohibition region is then transformed into an allowed velocity half-plane in the velocity plane. The intersection of all interaction pairs with their corresponding allowed velocity half-planes yields a joint allowed velocity region, where:

[0113] Construct a collision velocity prohibition region within the future prediction time window. Specifically, for each interaction pair, read the relative position and relative velocity, set the prediction time window to 2.5 seconds, set the collision determination radius to the sum of the equivalent radii of the two plus 0.20 meters, calculate the projection of the relative velocity in the direction of the relative position and the component of the relative velocity in the vertical direction, and calculate the nearest neighbor distance within 2.5 seconds. The nearest neighbor distance is the vertical distance from the relative position to the straight line in the direction of the relative velocity. When the nearest neighbor distance is less than or equal to the collision determination radius and the nearest moment is within the range of 0 to 2.5 seconds, it is determined that the current relative velocity will cause a collision. Define the set of relative velocities that meet the conditions as the collision velocity prohibition region, and divide the collision velocity prohibition region in the velocity plane with a separating straight line and take the side that does not contain the collision velocity prohibition region as the allowed velocity half-plane.

[0114] The joint allowable velocity domain is obtained by intersecting the allowable velocity half-planes corresponding to all interaction pairs. Specifically, the feasible velocity boundary of the unmanned forklift is used as the initial region. The feasible velocity boundary is determined by the linear velocity range of 0 to 2.0 meters per second and the angular velocity range of -1.0 to 1.0 radians per second. The allowable velocity half-plane of each interaction pair is clipped in turn. The part of the current region located on the allowable side is retained and forms a new convex polygon with the boundary line. When the clipped region is empty, the clipping of the current half-plane is canceled and the current half-plane is skipped. The convex polygon region obtained after all half-planes are processed is the joint allowable velocity domain.

[0115] By solving the reachability domain of vehicle motion, local control commands are selected within the joint permissible speed domain. The optimal action is then solved within the intersection of the reachability domain of vehicle motion and the joint permissible speed domain. The criterion for solving the optimal action is to minimize the sum of the square of the difference between the linear velocity and the reference velocity and the square of the difference between the angular velocity and the reference angular velocity. The linear velocity and angular velocity corresponding to the optimal action are then output as local control commands.

[0116] In this embodiment, the step of selecting the minimum cost trajectory, converting the minimum cost trajectory into a chassis control quantity, and issuing it for execution includes:

[0117] It receives local control commands and generates discrete control sequences. The discrete control sequence consists of multiple control steps. Each control step includes linear velocity commands and angular velocity commands. The time interval between control steps is a fixed control period.

[0118] Forward time-domain rolling simulation is performed on the discrete control sequence to obtain short-time trajectory clusters. The forward time-domain rolling simulation iteratively calculates the trajectory point sequence for each control step according to the vehicle kinematics update rule. The vehicle kinematics update rule is that the x-coordinate of the current position is equal to the x-coordinate of the previous moment plus the linear velocity multiplied by the cosine of the heading angle and then multiplied by the control period; the y-coordinate of the current position is equal to the y-coordinate of the previous moment plus the linear velocity multiplied by the sine of the heading angle and then multiplied by the control period; and the heading angle is equal to the heading angle of the previous moment plus the angular velocity and then multiplied by the control period.

[0119] The cost of calculating short-time trajectory clusters consists of trajectory smoothing cost, braking margin cost, and boundary fitting cost. The trajectory point sequence with the minimum cost is selected as the minimum cost trajectory. Specifically, the cost calculation for short-time trajectory clusters is as follows:

[0120] For each candidate trajectory, the linear velocity and angular velocity sequences and trajectory point coordinates are read point by point. The trajectory smoothing cost is calculated based on the difference in linear velocity and angular velocity between adjacent control steps. The squares of the linear velocity difference and the squares of the angular velocity difference at each step are summed and then summed over the entire trajectory. The braking margin cost is calculated based on the difference between the distance from the trajectory point to the nearest obstacle and the braking distance. The braking distance is obtained by dividing the square of the trajectory point's linear velocity by 2 and multiplying by the maximum deceleration. The maximum deceleration is taken as 2.0 m / s². When the distance to the nearest obstacle is less than the braking distance, the squares of the difference are accumulated. When it is greater than or equal to the distance, it is recorded as 0. The boundary fitting cost is calculated based on the lateral distance from the trajectory point to the global reference path. The squares of the lateral distance are summed over the entire trajectory. The three costs are directly added to obtain the total cost of the candidate trajectory. The trajectory point sequence with the minimum total cost is selected by traversing the short-time trajectory cluster.

[0121] The minimum cost trajectory is converted into chassis control quantities and issued for execution. The chassis control quantities include linear speed control quantities and steering control quantities. Execution feedback information is collected, which includes actual linear speed, actual steering status, and actual trajectory point sequence.

[0122] refer to Figure 3 An unmanned forklift obstacle avoidance scheduling system based on industrial vision includes the following modules:

[0123] The image acquisition and preprocessing module is used to acquire continuous image data from the vehicle-mounted industrial camera and output an image frame sequence after preprocessing.

[0124] An improved detection inference module is used to perform target detection on image frame sequences based on an improved RTMDet model and output the target detection results.

[0125] The cross-frame correlation modeling module is used to perform cross-frame correlation and velocity-acceleration continuity correction on target detection results, and output dynamic and static obstacle sets;

[0126] The traffic scheduling generation module is used to update the availability status of the passage section based on dynamic and static obstacle sets and output scheduling instruction information.

[0127] The collision avoidance decision module is used to perform local collision avoidance decisions based on the optimal reciprocal collision avoidance set of dynamic and static obstacles and scheduling instruction information, and output local control instructions.

[0128] The trajectory generation and execution module is used to generate discrete control sequences based on local control commands, perform forward time-domain rolling simulation to select the minimum cost trajectory, and then issue chassis control quantities for execution.

[0129] Example 1: To verify the feasibility of the present invention in practice, it was applied to a continuous mixed operation cycle in a warehouse workshop. The system received continuous image data collected by the industrial camera on the unmanned forklift to construct a training and validation dataset, consisting of 160 short sequences, each with 12 frames, totaling 1920 frames, with a resolution of 1280×720 and a frame rate of 20 frames per second. The target categories included pedestrians, unmanned forklifts, pallets, handcarts, and temporary barriers. Small targets accounted for 34.6%, occlusion accounted for 41.2%, and overexposed reflective areas accounted for 9.8%. The standard deviation of the mean grayscale fluctuation between frames was 18.4. The training set consisted of 900 frames, the validation set of 150 frames, and the test set of 102 frames, with a total of 1286 bounding boxes. In addition, 32 sequences in the test set were labeled with target identification numbers and trajectory ground truth values ​​for cross-frame association evaluation.

[0130] After the data enters the process, it is first preprocessed to generate an image frame sequence. After distortion correction, the average bending deviation of straight edge is reduced from 2.7 pixels to 0.6 pixels. Brightness normalization reads exposure and gain, performs exposure compensation, and normalizes it according to the mean and standard deviation of the whole frame, then maps it to 0 to 255 and truncates it. Before normalization, the grayscale mean range is 62 to 171, and after normalization, it converges to 118 to 136. The average difference between grayscale mean values ​​of adjacent frames is reduced from 14.8 to 6.1, and the saturated pixel ratio is reduced from 9.8% to 3.2%.

[0131] Subsequently, target detection training and testing were conducted. This invention uses the same training data and number of rounds as traditional methods; the traditional method uses a fixed-depth original RTMDet, while this invention uses an improved RTMDet. At training convergence, the validation set detection loss was 0.842 for the traditional method and 0.731 for this invention. The overall detection accuracy on the test set was 83.6% for the traditional method and 89.4% for this invention; the small target recall rate was 61.8% for the traditional method and 73.2% for this invention; the occluded target recall rate was 58.5% for the traditional method and 70.6% for this invention. During the inference phase, the percentage of early termination paths was 47.3% for shallow layers, 33.1% for mid-layers, and 19.6% for deep layers. The mean and standard deviation of detection latency were 38.6 ms and 8.9 ms for the traditional method, and 26.4 ms and 3.1 ms for this invention, respectively. The 95th percentile detection latency was 56.2 ms for the traditional method and 34.7 ms for this invention. The high-interference frame false negative rate was 12.7% for the traditional method and 6.1% for this invention.

[0132] In the cross-frame association stage, the center point of the detection box is transformed to a unified coordinate system of the stationary vehicle body, and a candidate association set is constructed with a distance threshold of 2.0 meters and a maximum of 20 candidates per point. The association cost tensor is generated according to geometric consistency, motion consistency, and appearance consistency, and a one-to-one global matching is performed. Matching costs greater than 0.60 are discarded. For ground truth evaluation of 320 trajectory segments, the number of target identity switching times is 24.3 times per 1000 frames using traditional methods, and 7.9 times using this invention; the mean absolute error of velocity estimation is 0.31 meters per second using traditional methods, and 0.18 meters per second using this invention; the proportion of velocity and acceleration peaks is reduced from 2.6% to 0.7%.

[0133] During the scheduling phase, the availability status of the passage segment is updated using dynamic and static obstacle sets, and a time-series occupancy table is maintained. Conflicts are determined when the overlap ratio of time windows is not less than 0.05. The waiting cost is calculated based on the maximum time the conflict ends plus a 0.5-second safety interval, multiplied by 1.0. The detour cost is calculated based on the detour distance increment multiplied by 0.2. High-density segment statistics show that out of every 100 passages, the traditional method had 17 instances of waiting exceeding 10 seconds, while this invention had 4. Deadlocks requiring forced replanning occurred 6 times with the traditional method, but 0 times with this invention. The average queue length at intersections was 2.8 vehicles with the traditional method, but 1.3 vehicles with this invention. The median congestion duration was 21.6 seconds with the traditional method, but 8.7 seconds with this invention.

[0134] Local collision avoidance employs an improved optimal reciprocal collision avoidance method. The interaction conditions are a relative distance not exceeding 12.0 meters and an approaching trend. The prediction time window is 2.5 seconds, and the collision determination radius is the sum of the equivalent occupied radii of both collisions plus 0.20 meters. The allowed velocity domain is clipped and intersected with an initial domain of linear velocity from 0 to 2.0 meters per second and an angular velocity from -1.0 to 1.0 radians per second. Traditional methods use a fixed 50 / 50 responsibility ratio and select the velocity first, then calculate the impact on the chassis, resulting in a higher rate of non-executable commands. The chassis limiting and roof clipping ratio is 14.6% for traditional methods and 3.9% for this invention. For close-range interactions with a minimum distance of less than 0.6 meters, the traditional method executes 5.2 collisions per hour, while this invention executes 1.4 collisions per hour.

[0135] In the trajectory generation and execution phase, short-time trajectory clusters are generated through rolling simulation of the discrete control sequence, and the smoothing cost, braking margin cost, and boundary contact cost are calculated. The trajectory with the minimum cost is selected, and the maximum deceleration is set to 2.0 m / s². Statistical analysis shows that the average change in trajectory angular velocity is 0.31 radians per second for the traditional method and 0.18 radians per second for this invention; the number of emergency stops is 4.5 times per hour for the traditional method and 0.8 times per hour for this invention; and the percentage of braking margins less than 0.8 m is 7.6% for the traditional method and 1.9% for this invention.

[0136] The comprehensive comparison results show that the overall detection accuracy is 83.6% for the traditional method and 89.4% for this invention; the small target recall rate is 61.8% for the traditional method and 73.2% for this invention; the identity switching frequency is 24.3 times per 1000 frames for the traditional method and 7.9 times for this invention; the average end-to-end decision latency is 112.5 milliseconds for the traditional method and 76.8 milliseconds for this invention, and the latency standard deviation is 19.4 milliseconds for the traditional method and 7.2 milliseconds for this invention; the average waiting time for narrow-channel vehicle encounters is 9.6 seconds for the traditional method and 3.1 seconds for this invention; the task volume per unit time is 31.4 units per hour for the traditional method and 38.9 units per hour for this invention; and the energy consumption per unit task is 0.41 kWh for the traditional method and 0.35 kWh for this invention. These results verify the improvements of this invention in perception stability, executability, and multi-vehicle traffic efficiency.

[0137] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for scheduling obstacle avoidance of unmanned forklifts based on industrial vision, characterized in that, include: Collect continuous image data from the industrial camera mounted on the unmanned forklift, preprocess the continuous image data to obtain an image frame sequence; An improved RTMDet model is constructed to perform target detection on image frame sequences. The segmented inference path selection is performed based on the segmented early retreat path to determine the inference depth. The scenario adaptive routing is performed using conditional computing power routing to determine the computation sub-path. The collaborative rearrangement is performed based on the main neck collaborative rearrangement to complete feature generation and fusion, and the target detection results are obtained. Cross-frame association processing is performed on the target detection results. A cross-frame candidate association set is constructed in the uniform coordinate system where the vehicle is stationary, and an association cost tensor is generated. Velocity-acceleration continuity correction is performed based on the association cost tensor, and dynamic and static obstacle sets are output. The availability status of the access sections in the work area is updated by dynamic and static obstacle sets. Look-ahead insertion and rollback verification are performed to identify conflict sections. The waiting cost and detour cost of the conflict sections are calculated to obtain scheduling instruction information. Based on optimal reciprocal collision avoidance, local collision avoidance decision processing is performed on dynamic and static obstacle sets and scheduling instruction information. The responsibility ratio is determined by a variable responsibility allocation component. The feasible action is solved by solving the reachability domain of vehicle motion and the optimal action is selected to obtain local control instructions. A discrete control sequence is generated based on the local control command, a forward time-domain rolling simulation is performed, the minimum cost trajectory is selected, the minimum cost trajectory is converted into a chassis control quantity and issued for execution.

2. The obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to claim 1, characterized in that, The continuous image data includes timestamp sequences, image frame pixel matrices, camera intrinsic data, camera extrinsic data, lens distortion parameter data, and camera exposure and gain parameter data.

3. The obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to claim 1, characterized in that, The obtained image frame sequence includes: The system collects continuous image data output from the industrial camera on the unmanned forklift. The continuous image data includes timestamp sequence, image frame pixel matrix, camera intrinsic parameter data, camera extrinsic parameter data, lens distortion parameter data, and camera exposure and gain parameter data. The continuous image data is time-aligned to form an image frame sequence sorted by timestamp. Distortion correction is performed on the image frame pixel matrix based on lens distortion parameter data, and brightness normalization is performed on the image frame pixel matrix based on camera exposure and gain parameter data. The distortion-corrected image frame pixel matrix and the brightness-normalized image frame pixel matrix are then unified into the target detection input format. Based on camera intrinsic and extrinsic data, coordinate parameter binding processing is performed on the image frame sequence to generate image frames associated with calibration information.

4. The obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to claim 1, characterized in that, The obtained target detection results include: An improved RTMDet model is constructed. The improved RTMDet model sets up a segmented early retreat path and a main-neck collaborative rearrangement structure between the backbone feature extraction structure and the neck feature fusion structure of the original RTMDet model. Conditional computing power routing is set up inside the original backbone feature extraction structure. The image frame sequence is processed by segmented early termination path to obtain the intra-frame target density index, intra-frame texture complexity index and inter-frame brightness change index of the image frame sequence, forming an inference depth discriminant. Based on the inference depth discriminant, the target early termination point is determined in the early termination point of the segmented early termination path. The target detection inference calculation is completed within the inference depth range corresponding to the target early termination point to obtain the target detection result. Conditional computing power routing processing is performed on the feature extraction process within the inference depth range corresponding to the early exit of the target. A scene routing identifier is generated based on the intermediate features corresponding to the early exit of the target. The target computing sub-path is determined in the parallel computing sub-path based on the scene routing identifier. Feature update and transmission are completed within the target computing sub-path to obtain routing features. The routing features are processed by main-neck collaborative rearrangement. A computational dependency table is established between each output node of the main feature extraction structure and each fusion node of the neck feature fusion structure. The computational execution order of the main feature extraction structure and the neck feature fusion structure is determined according to the computational dependency table, and the computational order is rearranged. Feature generation and feature fusion are completed according to the computational execution order to obtain fused features. The target detection results are output based on the fused features. The improved RTMDet model is trained by constructing a training sample set and batch inputting the training sample set, calculating the detection loss based on the bounding boxes and category labels of the training sample set, performing backpropagation to update the model parameters, and completing the training of the improved RTMDet model.

5. The obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to claim 1, characterized in that, The output set of dynamic and static obstacles includes: Based on the unmanned forklift wheel speed, rudder angle and timestamp sequence, the planar pose increment of the unmanned forklift between two adjacent frames is calculated. The center point of the bounding box in the target detection result is transformed according to the planar pose increment to obtain the set of detection points in the unified coordinate system of the stationary vehicle body. In a stationary, unified coordinate system of the vehicle body, a cross-frame candidate association set is constructed using the set of detection points as nodes, and an association cost tensor is generated for each candidate pairing relationship. Based on the association cost tensor, cross-frame global association is performed on the cross-frame candidate association set to obtain the target trajectory set. Based on the target trajectory set, the velocity sequence and acceleration sequence of each target trajectory at adjacent time points are calculated. Continuity correction is performed on the velocity sequence and acceleration sequence to obtain the target state information. Based on the target state information, the target trajectory set is divided into dynamic and static obstacle sets.

6. The obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to claim 1, characterized in that, The obtained scheduling instruction information includes: A set of access sections is generated based on the work area map, and the availability status of each access section is determined. Receive the task queue and the current status of the unmanned forklift and generate an initial passage plan. The initial passage plan includes the sequence of passage sections and the corresponding time windows. Write the initial passage plan into the time-series occupancy table. The time-series occupancy table records the time windows occupied by the unmanned forklift in each passage section, with the passage section as the index. The initial passage plan is inserted forward and conflict sections are identified. The time window to be inserted is compared with the existing time window of the same passage section in the time-series occupancy table. The passage section that meets the judgment condition is identified as a conflict section. The conflict section is rolled back and verified to generate a set of alternative passage plan candidates. Calculate the waiting cost and detour cost of the conflict section, select the alternative passage plan with the minimum sum of waiting cost and detour cost from the alternative passage plan candidate set as the scheduling result, and output the scheduling instruction information including the passage section sequence, the occupied time window and the waiting or detour instructions.

7. The obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to claim 1, characterized in that, The obtained local control commands include: The system receives information on the unmanned forklift's own status, dynamic obstacle set, and scheduling instructions. The unmanned forklift's own status includes its current position, current heading, current linear velocity, and current turning state. Each dynamic obstacle in the dynamic obstacle set includes its position, velocity, and equivalent occupied radius. A variable responsibility allocation component is used to construct an interaction pair between the unmanned forklift and each dynamic obstacle. A responsibility ratio parameter is calculated for each interaction pair. The responsibility ratio parameter is used to determine the avoidance share of the unmanned forklift in the interaction pair. Based on the relative position and relative speed of the unmanned forklift and dynamic obstacles, a collision speed prohibition region is constructed within the future prediction time window. The collision speed prohibition region is transformed into an allowed speed half-plane in the speed plane. The intersection of all interaction pairs with the corresponding allowed speed half-planes is obtained to obtain the joint allowed speed region. By solving the reachability domain of vehicle motion, local control commands are selected within the joint permissible speed domain. The optimal action is then solved within the intersection of the reachability domain of vehicle motion and the joint permissible speed domain. The criterion for solving the optimal action is to minimize the sum of the square of the difference between the linear velocity and the reference velocity and the square of the difference between the angular velocity and the reference angular velocity. The linear velocity and angular velocity corresponding to the optimal action are then output as local control commands.

8. The obstacle avoidance scheduling method for unmanned forklifts based on industrial vision according to claim 1, characterized in that, The step of selecting the minimum cost trajectory, converting the minimum cost trajectory into a chassis control quantity, and issuing it for execution includes: It receives local control commands and generates discrete control sequences. The discrete control sequence consists of multiple control steps. Each control step includes linear velocity commands and angular velocity commands. The time interval between control steps is a fixed control period. Forward time-domain rolling simulation is performed on the discrete control sequence to obtain short-time trajectory clusters. The forward time-domain rolling simulation iteratively calculates the trajectory point sequence for each control step according to the vehicle kinematics update rule. The vehicle kinematics update rule is that the x-coordinate of the current position is equal to the x-coordinate of the previous moment plus the linear velocity multiplied by the cosine of the heading angle and then multiplied by the control period; the y-coordinate of the current position is equal to the y-coordinate of the previous moment plus the linear velocity multiplied by the sine of the heading angle and then multiplied by the control period; and the heading angle is equal to the heading angle of the previous moment plus the angular velocity and then multiplied by the control period. The cost is calculated for short-time trajectory clusters. The cost consists of trajectory smoothing cost, braking margin cost, and boundary fitting cost. The trajectory point sequence with the minimum cost is selected as the minimum cost trajectory. The minimum cost trajectory is converted into chassis control quantities and issued for execution. The chassis control quantities include linear speed control quantities and steering control quantities. Execution feedback information is collected, which includes actual linear speed, actual steering status, and actual trajectory point sequence.

9. An unmanned forklift obstacle avoidance scheduling system based on industrial vision, comprising performing the method described in any one of claims 1 to 8, characterized in that, Includes the following modules: The image acquisition and preprocessing module is used to acquire continuous image data from the vehicle-mounted industrial camera and output an image frame sequence after preprocessing. An improved detection inference module is used to perform target detection on image frame sequences based on an improved RTMDet model and output the target detection results. The cross-frame correlation modeling module is used to perform cross-frame correlation and velocity-acceleration continuity correction on target detection results, and output dynamic and static obstacle sets; The traffic scheduling generation module is used to update the availability status of the passage section based on dynamic and static obstacle sets and output scheduling instruction information. The collision avoidance decision module is used to perform local collision avoidance decisions based on the optimal reciprocal collision avoidance set of dynamic and static obstacles and scheduling instruction information, and output local control instructions. The trajectory generation and execution module is used to generate discrete control sequences based on local control commands, perform forward time-domain rolling simulation to select the minimum cost trajectory, and then issue chassis control quantities for execution.

Images