A forklift illegal behavior AI identification method and system

Abstract

The application discloses a forklift illegal behavior AI identification method and system, belongs to the computer vision and artificial intelligence technical field, and includes: acquiring a continuous video frame of a warehouse operation area and performing adaptive image preprocessing; inputting an enhanced video frame sequence into a forklift special lightweight multi-target detection network trained by small sample enhancement, and outputting a multi-target detection result; performing time sequence trajectory fitting on continuous multi-frame detection results, and generating a time sequence motion feature vector; combining a static visual feature with a local RAG knowledge base to perform rule matching reasoning, and outputting an illegal judgment result; and generating a graded alarm signal and an encrypted evidence record according to an illegal risk level. The application combines lightweight multi-target detection and time sequence trajectory analysis, can realize accurate identification of various illegal behaviors in forklift operation, and significantly improves the intelligent level and real-time performance of warehouse operation safety supervision.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to an AI-based method and system for identifying forklift violations. Background Technology

[0002] Forklifts, as core operating equipment in modern logistics warehousing systems, directly impact the overall operational efficiency and personnel safety of the warehousing system through their operational efficiency and safety management level. Utilizing artificial intelligence technologies, particularly computer vision and image analysis, for real-time monitoring and violation identification of forklift operations is a crucial direction for improving the intelligence level of warehouse safety management. This technology analyzes video streams captured by cameras to automatically detect key targets such as forklifts, personnel, and goods, and determines whether their behavior complies with safety regulations, thereby achieving proactive early warning and post-event traceability.

[0003] Existing forklift safety monitoring technologies typically employ general-purpose multi-object detection networks to analyze video images and identify targets such as forklifts and personnel. After acquiring the pixel coordinates and outline dimensions of the targets, these systems determine whether violations exist by setting fixed geometric positional relationships or simple motion threshold rules. For example, they might warn of collision risks by determining the distance between the personnel bounding box and the forklift bounding box, or estimate the speed by calculating the forklift's displacement between consecutive frames, thereby determining whether it is speeding.

[0004] However, existing technologies still have several technical shortcomings in practical applications. First, warehouse environments are characterized by variable lighting, dust, and frequent partial obstructions from goods and shelves. General image analysis methods are insufficiently adaptable to these environmental changes, leading to decreased target detection accuracy. Second, targets in forklift operation scenarios vary significantly in size, such as large forklift vehicles and thin seat belts, making it difficult for standard detection networks to cover all aspects and easily resulting in missed detections of small targets. Furthermore, simple logical rules are insufficient to accurately describe complex violations, and these rules are fixed in the code, making updates and maintenance inconvenient and unable to flexibly adapt to the customized management regulations of different parks. Finally, most solutions rely on cloud servers for computation, and network latency and instability can affect the real-time performance of alarms. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an AI-based method and system for identifying forklift violations. It employs enhanced multi-target detection and temporal feature extraction from video frame sequences, combined with vectorized safety specifications for semantic matching and logical reasoning. This enables automated, high-precision identification and tiered alerting of forklift operation violations.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, a method for AI-based identification of forklift violations is provided, comprising: acquiring continuous video frames of a warehouse operation area; performing adaptive image preprocessing on the continuous video frames to obtain an enhanced video frame sequence; inputting the enhanced video frame sequence into a lightweight multi-object detection network specifically designed for forklifts, trained with few-sample enhancement, performing multi-scale feature fusion and simultaneous detection of multiple target classes, and outputting the multi-object detection result for each video frame, wherein the multi-object detection result includes pixel coordinates, contour dimensions, and category confidence labels of the forklift body, cargo pallet, seat belt, driver's human outline, forks, and surrounding pedestrians; performing temporal trajectory fitting on the multi-object detection results of multiple consecutive frames, extracting temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver's limb posture changes, and distance between pedestrians and the fork area, and generating a temporal motion feature vector. The system acquires the text of forklift safety regulations and the text of customized management regulations for the industrial park. It then performs text vectorization on these texts to generate a vectorized safety regulation index, which is stored in a local RAG knowledge base. The system extracts the target category and contour size from the multi-target detection results as static visual features. These static visual features and the temporal motion feature vector are input into a rule matching inference unit. The rule matching inference unit retrieves the vectorized safety regulation index from the local RAG knowledge base, performs semantic matching, and combines the static visual features with the temporal motion feature vector to output a violation determination result. Based on the violation determination result, the system determines the violation risk level, generates a graded alarm signal corresponding to the violation risk level, extracts key frame images of the violation and short video streams of the corresponding time period, and adds metadata to generate an encrypted evidence collection ledger record.

[0008] Based on the above technical solution, the AI-based method for identifying forklift violations provided in this application employs enhanced multi-target detection and temporal feature extraction of video frame sequences, combined with vectorized safety specifications for semantic matching and logical reasoning, which enables automated, high-precision identification and graded alarms for forklift operation violations.

[0009] In conjunction with the first aspect above, in one possible implementation, acquiring continuous video frames of the warehousing operation area and performing adaptive image preprocessing on the continuous video frames includes: performing layered dynamic brightness equalization on the continuous video frames, automatically adjusting the brightness of the image based on the pixel grayscale distribution of a single frame to obtain a brightness correction frame; performing local occlusion repair on the brightness correction frame, detecting occluded pixel areas and using the spatiotemporal information of adjacent frames to fill the occluded areas with pixels to obtain an occlusion repair frame; and performing fork motion distortion correction on the occlusion repair frame, performing geometric transformation correction based on the degree of distortion of the fork edge contour to generate an enhanced video frame sequence.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the lightweight multi-target detection network for forklifts trained with few-sample augmentation is reconstructed based on the YOLO backbone network, including: adding a first cross-scale feature fusion branch to the feature fusion part of the network to fuse multi-level feature maps of large forklift body targets and small seat belt targets; adding a second cross-scale feature fusion branch to fuse multi-level feature maps of medium-sized cargo pallet targets and slender fork targets; and concatenating the feature maps output by the first and second cross-scale feature fusion branches with the original feature maps, and inputting them together into the detection head to output multi-target detection results.

[0011] In conjunction with the first aspect above, in one possible implementation, the training sample set of the forklift-specific lightweight multi-object detection network includes: acquiring real sample images of forklift operations in logistics and warehousing parks, and labeling the forklift body, cargo pallets, seat belts, driver's human silhouette, forks, and pedestrian targets with categories and bounding boxes to obtain real labeled samples; performing mask transformation and dynamic cargo overlay on the real labeled samples to generate enhanced samples under cargo occlusion conditions; combining illumination simulation to generate illumination-enhanced samples under different illumination conditions, and mixing the real labeled samples, enhanced samples, and illumination-enhanced samples to form a small sample enhanced training set for training the forklift-specific lightweight multi-object detection network.

[0012] In conjunction with the first aspect above, in one possible implementation, the step of performing temporal trajectory fitting on the multi-target detection results of consecutive frames to extract temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver limb posture changes, and the distance between pedestrians and the fork area includes: smoothing the pixel coordinates of each target in consecutive frames using Kalman filtering to obtain the motion trajectory curve of each target; calculating the forklift speed and steering angle based on the inter-frame displacement and orientation angle changes of the forklift body target on the motion trajectory curve; calculating the fork lifting displacement amplitude based on the vertical coordinate changes of the fork target on the motion trajectory curve; extracting driver limb posture change features based on the inter-frame coordinate changes of the skeletal key points within the driver's human body contour; calculating the distance features between pedestrians and the fork area based on the inter-frame changes of the coordinates of the pedestrian target and the boundary of the fork area; and combining the forklift speed, steering angle, fork lifting displacement amplitude, driver limb posture change features, and distance features between pedestrians and the fork area to form a temporal motion feature vector.

[0013] In conjunction with the first aspect above, in one possible implementation, the rule matching reasoning unit retrieves the vectorized safety specification index in the local RAG knowledge base for semantic matching and outputs a violation determination result by combining static visual features and temporal motion feature vectors. This includes: converting the target category and contour size contained in the static visual features, and the motion parameters contained in the temporal motion feature vectors, into a query statement describing the current operation status; using the query statement to perform vector similarity retrieval in the vectorized safety specification index to recall multiple candidate safety specification clauses; reordering the recalled candidate safety specification clauses and selecting the one with the highest similarity as the matching clause; and logically comparing the constraints specified in the matching clause with the static visual features and temporal motion feature vectors to generate a violation determination result.

[0014] In conjunction with the first aspect mentioned above, in one possible implementation, the constraints specified in the matching clause are logically compared with the static visual features and temporal motion feature vectors to generate a violation determination result. This includes: when the constraints of the matching clause determine that the following conditions occur: driver not wearing seat belt properly, forklift carrying passengers illegally, personnel standing on fork pallets, goods exceeding height or width and obstructing the driver's view, single fork carrying heavy loads, forklift raising and lowering during driving, speeding or driving against traffic in the factory area, or pedestrians entering the dangerous area of ​​fork operation, the corresponding violation type identifier and violation confidence level are output.

[0015] In conjunction with the first aspect mentioned above, in one possible implementation, the violation risk level is determined based on the violation judgment result, a graded alarm signal corresponding to the violation risk level is generated, key frame images of the violation and short video streams of the corresponding time period are captured, and metadata is added to generate an encrypted evidence collection ledger record. This includes: classifying the violation risk level into three levels—mild, moderate, and severe—based on the violation judgment result; when a severe violation is judged, a live audio-visual alarm command is generated simultaneously, a pop-up reminder command is pushed to the forklift vehicle terminal, and an alarm message is sent to the safety management backend, and a handling work order is automatically generated; when a moderate violation is judged, a work order to be handled is generated and pushed to the area administrator account; when a mild violation is judged, a violation record mark is generated and stored in the backend according to category; key frame images and short video streams within the time period corresponding to the violation judgment result are captured, and metadata including the park location number, forklift equipment identifier, violation occurrence timestamp, and violation behavior type identifier is added, encrypted, and stored to form an unalterable evidence collection ledger record.

[0016] In conjunction with the first aspect above, in one possible implementation, the method is deployed on an edge computing gateway: the edge computing gateway has a built-in image preprocessing module, an AI detection and inference module, a time-series trajectory analysis module, a local RAG knowledge base, and a rule matching and inference module; when the network connection between the edge computing gateway and the cloud management platform is interrupted, the edge computing gateway continues to execute the steps and caches the generated hierarchical alarm signals and encrypted evidence collection ledger records, and automatically synchronizes the cached data to the cloud management platform in batches after the network connection is restored.

[0017] Secondly, an AI-based forklift violation recognition system is provided, comprising: a video acquisition and preprocessing module for acquiring continuous video frames of a warehouse operation area, performing adaptive image preprocessing on the continuous video frames to obtain an enhanced video frame sequence; a multi-target synchronous detection module for inputting the enhanced video frame sequence into a lightweight multi-target detection network specifically designed for forklifts, trained with few-sample enhancement, performing multi-scale feature fusion and multi-class target synchronous detection, and outputting the multi-target detection result for each video frame, wherein the multi-target detection result includes pixel coordinates, contour dimensions, and category confidence labels of the forklift body, cargo pallet, seat belt, driver's human contour, forks, and surrounding pedestrians; and a temporal trajectory fitting and feature extraction module for performing temporal trajectory fitting on the multi-target detection results of multiple consecutive frames, extracting temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver's limb posture changes, and distance between pedestrians and the fork area, and generating... The system includes a temporal motion feature vector and a rule-matching inference module. The module acquires forklift safety regulations and park-customized management regulations, performs text vectorization on these texts, generates a vectorized safety regulation index, and stores it in a local RAG knowledge base. It extracts the target category and contour size from the multi-target detection results as static visual features. The static visual features and the temporal motion feature vector are input into the rule-matching inference unit. The rule-matching inference unit retrieves the vectorized safety regulation index from the local RAG knowledge base, performs semantic matching, and combines the static visual features and the temporal motion feature vector to output a violation judgment result. A graded alarm and evidence collection record module determines the violation risk level based on the violation judgment result, generates a graded alarm signal corresponding to the violation risk level, captures key frame images of the violation and short video streams of the corresponding time period, and adds metadata to generate an encrypted evidence collection ledger record.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] This invention constructs an adaptive image preprocessing pipeline that includes layered dynamic brightness equalization, spatiotemporal restoration of local occlusion, and forklift motion distortion correction. This effectively suppresses image degradation caused by sudden changes in illumination, occlusion from stacked goods, and rapid forklift movement in warehousing environments, significantly enhancing the quality and information integrity of input images. By combining few-sample augmentation training strategies such as mask transformation, dynamic goods overlay, and illumination simulation, diverse training data covering various occlusion modes and illumination conditions is generated under limited real-labeled samples, enabling the detection network to fully learn the visual diversity features of forklift operation scenarios. The synergistic effect of preprocessing enhancement and data augmentation significantly improves the robustness and generalization ability of the detection model in complex warehousing environments for detecting multiple targets such as forklift bodies, slender forks, small-sized seat belts, and pedestrians, reducing false positives and false negatives caused by environmental changes.

[0020] This invention specifically reconstructs the YOLO backbone network, adding a dual-scale feature fusion branch to the feature fusion layer. This branch adapts to the differentiated feature representation needs of large forklift bodies and small seat belts, as well as medium-sized pallets and slender forks. Through the full fusion and stitching of multi-level feature maps, the detection head simultaneously acquires rich global semantic information and local detail information, achieving synchronous high-precision detection of six types of targets with significant size differences. Furthermore, Kalman filtering and temporal trajectory fitting are applied to the detection results of multiple consecutive frames. This accurately extracts multi-dimensional temporal motion features from the pixel-level motion trajectory, including forklift speed, steering angle, fork lifting displacement, driver posture changes, and pedestrian-fork distance. This elevates the static spatial detection results to a dynamic behavior understanding level, providing complete, continuous, and accurate motion state data for violation identification.

[0021] This invention extracts target categories and contour dimensions from multi-target detection results to form static visual features, which, together with the aforementioned temporal motion feature vectors, constitute a comprehensive digital description of the current work scenario. Based on this, forklift safety regulations and customized park management regulations are vectorized to construct a local RAG knowledge base. Through query statement vector retrieval and candidate clause reordering mechanisms, the most semantically similar regulatory clauses most relevant to the current work status are automatically matched. Furthermore, the constraints contained in the clauses are compared in a structured logical manner with the static visual features and temporal motion features to accurately determine eight typical violations: improper seatbelt use, unauthorized forklift carrying of passengers, personnel standing on forks and pallets, goods exceeding height or width obstructing the view, carrying heavy loads on a single fork, raising and lowering forks while in motion, speeding, driving against traffic, and pedestrians entering dangerous areas. The introduction of the local RAG knowledge base allows park managers to dynamically expand violation judgment rules simply by adding, deleting, or modifying regulatory text, without retraining the model, thus improving the system's adaptability and deployment flexibility under different park conditions and management requirements.

[0022] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims, and drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A structural architecture diagram of a forklift violation AI recognition system provided in this application embodiment;

[0025] Figure 2 A flowchart illustrating an AI-based method for identifying forklift violations provided in this application embodiment;

[0026] Figure 3 This is a multi-scale target spatial size distribution model diagram provided in the embodiments of this application.

[0027] Figure 4 This is a distribution ratio diagram of the small sample augmented training set data provided in the embodiments of this application. Detailed Implementation

[0028] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] The forklift violation AI recognition method and system provided in this application embodiment can be applied to, for example... Figure 1 As shown in the AI-based forklift violation recognition system, such as Figure 1As shown, the system includes: a video acquisition and preprocessing module, used to acquire continuous video frames of the warehousing operation area, and perform adaptive image preprocessing on the continuous video frames to obtain an enhanced video frame sequence; a multi-target synchronous detection module, used to input the enhanced video frame sequence into a lightweight multi-target detection network specifically designed for forklifts trained with few-sample enhancement, to perform multi-scale feature fusion and multi-class target synchronous detection, and output the multi-target detection result for each video frame, wherein the multi-target detection result includes the pixel coordinates, contour size, and category confidence labels of the forklift body, cargo pallet, seat belt, driver's human contour, forks, and surrounding pedestrians; and a temporal trajectory fitting and feature extraction module, used to perform temporal trajectory fitting on the multi-target detection results of multiple consecutive frames, extract temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver's limb posture changes, and distance between pedestrians and the fork area, and generate a temporal motion feature vector. The rule matching and reasoning module is used to acquire forklift safety regulations text and park customized management regulations text, perform text vectorization processing on the forklift safety regulations text and park customized management regulations text, generate a vectorized safety regulations index and store it in the local RAG knowledge base, extract the target category and contour size from the multi-target detection results as static visual features, input the static visual features and the temporal motion feature vector into the rule matching and reasoning unit, the rule matching and reasoning unit retrieves the vectorized safety regulations index in the local RAG knowledge base, performs semantic matching and combines the static visual features and temporal motion feature vector to output the violation judgment result; the graded alarm and evidence collection record module is used to determine the violation risk level based on the violation judgment result, generate graded alarm signals corresponding to the violation risk level, capture violation key frame images and corresponding short video streams, attach metadata to generate encrypted evidence collection ledger records.

[0030] like Figure 2As shown in the embodiments of this application, an AI-based method for identifying forklift violations is provided, comprising: acquiring continuous video frames of a warehouse operation area; performing adaptive image preprocessing on the continuous video frames to obtain an enhanced video frame sequence; inputting the enhanced video frame sequence into a lightweight multi-object detection network specifically designed for forklifts, trained with few-sample enhancement, performing multi-scale feature fusion and simultaneous detection of multiple target classes, and outputting the multi-object detection result for each video frame, wherein the multi-object detection result includes pixel coordinates, contour dimensions, and category confidence labels of the forklift body, cargo pallet, seat belt, driver's human body contour, forks, and surrounding pedestrians; performing temporal trajectory fitting on the multi-object detection results of multiple consecutive frames, extracting temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver's limb posture changes, and distance between pedestrians and the fork area, and generating temporal motion features. Vectorization is performed on the forklift safety regulations text and the park's customized management regulations text. The forklift safety regulations text and the park's customized management regulations text are then vectorized to generate a vectorized safety regulations index, which is stored in a local RAG knowledge base. The target category and contour size from the multi-target detection results are extracted as static visual features. The static visual features and the temporal motion feature vectors are input into a rule matching inference unit. The rule matching inference unit retrieves the vectorized safety regulations index from the local RAG knowledge base, performs semantic matching, and combines the static visual features and temporal motion feature vectors to output a violation judgment result. Based on the violation judgment result, the violation risk level is determined, and a graded alarm signal corresponding to the violation risk level is generated. Key frame images of the violation and short video streams of the corresponding time period are captured, and metadata is appended to generate an encrypted evidence collection ledger record.

[0031] Optionally, the step of acquiring continuous video frames of the warehousing operation area and performing adaptive image preprocessing on the continuous video frames includes: performing layered dynamic brightness equalization on the continuous video frames, automatically adjusting the brightness of the image based on the pixel grayscale distribution of a single frame to obtain a brightness correction frame; performing local occlusion repair on the brightness correction frame, detecting occluded pixel areas and using the spatiotemporal information of adjacent frames to fill the occluded areas with pixels to obtain an occlusion repair frame; and performing fork motion distortion correction on the occlusion repair frame, performing geometric transformation correction based on the degree of distortion of the fork edge contour to generate an enhanced video frame sequence.

[0032] Specifically, layered dynamic brightness equalization is performed on consecutive video frames, automatically adjusting the brightness of the image based on the pixel grayscale distribution of each frame to obtain a brightness-corrected frame. Each frame is divided into several sub-regions, and histogram equalization with contrast limits is performed independently on each sub-region. The final output grayscale value of any pixel is obtained by bilinear interpolation of the equalization results of its own sub-region and adjacent sub-regions.

[0033] ;

[0034] in, Representing coordinates The final grayscale value of the pixel in the brightness-corrected frame. , , , These represent the gray values ​​calculated by the gray-scale mapping function after histogram equalization of the four nearest sub-region centers containing the pixel. and It is the normalized horizontal and vertical distance from the pixel to the center of its top-left sub-region, with a value range of... This is used for bilinear interpolation weight calculation. Local occlusion region repair is performed on the brightness-corrected frame by detecting occluded pixel regions and filling them with spatiotemporal information from adjacent frames to obtain the occlusion-repaired frame. For any pixel within the occluded region, the repair value is obtained by weighted fusion of pixels at the corresponding position in temporally adjacent, unoccluded frames.

[0035] ;

[0036] in, It is the current number The coordinates in the frame are The pixel values ​​to be repaired. and They are the first Frame and the The pixel value of the frame. and These are the optical flow vectors traced backward from the current frame to the two frames before and after. and These are weighting coefficients, and Its value is dynamically adjusted based on the confidence level of the optical flow vector, with frames having higher confidence levels receiving higher weights. Fork motion distortion correction is performed on the occlusion repair frames, and perspective transformation correction is applied based on the degree of distortion of the fork edge contours to generate an enhanced video frame sequence. The degree of distortion is quantified by the deviation distance between the intersection of the two straight lines on the fork edge and the image centerline. The correction uses the following perspective transformation:

[0037] ;

[0038] in, These are the original pixel coordinates in the occlusion repair frame, while the corrected coordinates are... . It is the perspective transformation matrix calculated based on the degree of distortion. (Matrix) Eight unknown parameters to This is achieved by solving a set of linear equations derived from the coordinates of the vertices of the distorted contour and their corresponding points in a parallel view of the target. Applying this transformation to the entire occlusion restoration frame generates the final enhanced video frame sequence.

[0039] For example, suppose the system acquires a continuous video frame with a resolution of 1280×720 pixels. The image is divided into 64 sub-regions of 8×8 pixels. Consider the pixels located at the boundaries between two sub-regions. The gray values ​​of its four nearest sub-regions after center equalization are respectively , , , Normalized distance , Then the grayscale value of this pixel in the brightness correction frame is: The value, rounded to 99, increases the brightness of that point. In occlusion repair, the coordinates... The pixels were identified as part of the occluded region. The system calculated the optical flow vector traced back to the previous frame as follows: The optical flow vector in the next frame is The position corresponding to the previous frame. The pixel value is 150, and the corresponding position in the next frame. The pixel value is 160, and the weighting coefficient is... , Then the pixel repair value is: ; blemished pixels are smoothed and repaired. Fork motion distortion correction is performed on the occlusion-repaired frames. The coordinates of the four vertices of the detected fork edge contour are as follows: , , , It presents a trapezoidal shape. The goal is to correct it into a rectangle with a width of 400 pixels and a height of 300 pixels, with the target vertex being... , , , The perspective transformation matrix is ​​calculated as follows: Transform the original point (800, 320): The corrected coordinates are After transforming all pixels of the entire frame image, an enhanced video frame sequence is finally generated, in which fork distortion is effectively corrected.

[0040] Optionally, the lightweight multi-target detection network for forklifts trained with few-sample enhancement is reconstructed based on the YOLO backbone network, including: adding a first cross-scale feature fusion branch to the feature fusion part of the network to fuse multi-level feature maps of large forklift body targets and small seat belt targets; adding a second cross-scale feature fusion branch to fuse multi-level feature maps of medium-sized cargo pallet targets and slender fork targets; and concatenating the feature maps output by the first and second cross-scale feature fusion branches with the original feature maps, and inputting them together into the detection head to output multi-target detection results.

[0041] Specifically, a single-frame enhanced video frame is input into a lightweight multi-object detection network specifically for forklifts. A first cross-scale feature fusion branch is added to the network's feature fusion section to fuse multi-level feature maps of large forklift body targets and small seatbelt targets. This branch upsamples the deep feature maps to align their spatial dimensions with the shallow feature maps, then concatenates them along the channel dimension, and extracts information through convolutional layers. The output can be represented as:

[0042] ;

[0043] in, The feature map represents the output of the first cross-scale feature fusion branch. It is the deep raw feature map that is relied upon for detecting large targets. It is a shallow, raw feature map that is relied upon for detecting small targets. This indicates an upsampling operation, such as bilinear interpolation, used for amplification. Spatial dimension. To represent a transformation operation, usually Convolution, used for adjustment The number of channels to match the upsampled . This represents a splicing operation along the channel dimension. The subsequent convolutional operation is used to fuse the concatenated features and reduce the channel dimension. To resolve the detection conflict between medium-sized cargo pallets and slender forklift targets, the network adds a second cross-scale feature fusion branch. This branch fuses two medium-level original feature maps, one focusing on expressing the overall shape of the object, and the other better at capturing slender structures. Its computation process is similar to the first branch, fusing multi-level feature maps optimized for these two specific target types. The feature maps output from the first and second cross-scale feature fusion branches are concatenated with the original feature maps. This means that the feature maps input to the final detection part not only contain general multi-scale information but also additionally inject optimized, higher-information-density feature representations for specific target pairs. These highly fused feature maps are input to the detection head, a small network responsible for predicting possible targets at each spatial location of the feature maps. The output of the detection head is the multi-target detection result for each video frame. For each successfully identified target in the frame, the result contains four core pieces of information: pixel coordinates, typically represented as the top-left corner of the target's bounding box. Coordinates; outline dimensions, i.e., the width of the bounding box. and height The category confidence label is a value between 0 and 1, representing the probability that the detection result belongs to a specific category. The network can simultaneously output target categories including forklift body, cargo pallet, seat belt, driver's silhouette, forks, and surrounding pedestrians. The final detection results... It can be represented as a tuple ,in These are bounding box parameters. It is a category. This is the category confidence label. The category confidence label is calculated as follows:

[0044] ;

[0045] in, The network determines the probability of any target existing within the bounding box, while The network determines the category of a target based on the premise that the target exists. The conditional probability. Only when The detection result is considered valid only when it exceeds a preset threshold. This threshold, for example, 0.5, is determined based on a precision-recall tradeoff analysis performed on a validation set containing 5000 images. Figure 3The diagram illustrates the size distribution differences of various detection targets in pixel space during warehousing operations. Due to the significant size difference between the massive vehicle body and the tiny seatbelt, and the different features of the medium-sized pallet and the slender forklift, standard detection networks struggle to accommodate them all. This graphical model demonstrates the necessity of adding first and second cross-scale feature fusion branches to the network to achieve simultaneous, high-precision detection of six types of targets with vastly different sizes.

[0046] For example, when an enhanced video frame of 1280×720 pixels is input into the network, the backbone network generates three layers of original feature maps: The dimensions are 320×180. The dimensions are 160×90. The dimensions are 80×45. The first cross-scale feature fusion branch is selected. and By upsampling by 4 times Upgraded from 80×45 to 320×180, adjusted via 1×1 convolution. After increasing the number of channels, the two are concatenated to generate a feature map. Selection of the second cross-scale feature fusion branch and Perform similar processing to generate .Will , , After stitching, the data is input into the detection head, which analyzes the data and outputs the target result. Taking a seatbelt target as an example, the output is the pixel coordinates. Outline dimensions The category is seat belt, and its category confidence label is calculated as follows: The value was higher than the preset threshold of 0.5 and was successfully detected. The output also included other targets such as the forklift body, cargo pallets, and driver's silhouette, collectively forming the multi-target detection result for this frame.

[0047] Optionally, the training sample set of the forklift-specific lightweight multi-object detection network includes: acquiring real sample images of forklift operations in logistics and warehousing parks, and labeling the forklift body, cargo pallets, seat belts, driver's human outline, forks, and pedestrian targets with categories and bounding boxes to obtain real labeled samples; performing mask transformation and dynamic cargo overlay on the real labeled samples to generate enhanced samples under cargo occlusion conditions; combining illumination simulation to generate illumination-enhanced samples under different illumination conditions, and mixing the real labeled samples, enhanced samples, and illumination-enhanced samples to form a small sample enhancement training set for training the forklift-specific lightweight multi-object detection network.

[0048] Specifically, real sample images of forklift operations in logistics and warehousing parks are collected, and the forklift body, cargo pallets, seat belts, driver's human silhouette, forks, and pedestrian targets are categorized and labeled with bounding boxes to obtain real labeled samples. Mask transformation and dynamic cargo overlay are applied to the real labeled samples to generate enhanced samples under cargo occlusion conditions. Mask transformation generates a binary mask image of the targets in the real labeled samples, with a value of 1 in the target area and a value of 0 in the background area; dynamic cargo overlay randomly overlays the extracted targets onto the forklift or forks of another real labeled sample. The pixel values ​​of the enhanced samples are calculated as follows:

[0049] ;

[0050] in, The coordinates in the generated enhanced sample are The pixel value at that location. These are the pixel values ​​of the actual labeled samples used as background at the same coordinates. The target image extracted from another real-world labeled sample is located at the corresponding coordinates. The pixel value. The mask image corresponding to the target is in The pixel value at each location is either 1 or 0. This formula ensures that background pixels are replaced only with target pixels in areas where the mask value is 1, thus achieving seamless overlay. Warehouse environments experience drastic changes in lighting conditions. To improve the network's adaptability to these changes, it's necessary to generate lighting-enhanced samples under different illumination conditions using lighting simulation. This process simulates changes in brightness and contrast by linearly transforming the pixel values ​​of the real labeled samples. For each pixel in the image, its transformed pixel value is calculated as follows:

[0051] ;

[0052] in, The generated illumination-enhanced sample in coordinates The pixel value at that location. These are the pixel values ​​of the original, real-world labeled samples at the same coordinates. It is the contrast adjustment factor. These are brightness adjustment factors. These two factors are randomly selected from a preset range each time the image is generated. For example, from Choose from, from The ranges were selected based on statistical analysis of brightness and contrast in over 500 real-world scene images under different lighting conditions. This allows for the simulation of various lighting effects from dawn to dusk, and from sunny to cloudy days. The original ground truth labeled samples, enhanced samples generated through mask transformation and dynamic cargo overlay, and lighting-enhanced samples generated through lighting simulation are mixed. This mixed dataset, the few-shot augmentation training set, is used to train a lightweight multi-object detection network specifically for forklifts. Because this training set includes not only real-world scenes but also a large number of simulated occlusion and lighting variation scenarios, the trained network model possesses stronger generalization ability and robustness. Figure 4 The diagram illustrates the specific data composition of the small-sample augmented training set used to train the lightweight multi-object detection network for forklifts. By mixing the initial 2000 ground-labeled images with 4000 images of cargo occlusion augmentation generated through mask transformation and dynamic cargo overlay, and 4000 images of illumination augmentation generated through lighting simulation, a high-quality training set of 10,000 images was ultimately formed. This diagram visually reflects the data foundation of the synergistic effect of preprocessing augmentation and data augmentation.

[0053] For example, initially collect 2000 real sample images. Select one real labeled sample image A containing an empty forklift, labeled as: forklift body. Forklift Select another real-world labeled sample image B containing a single cargo pallet, labeled as: cargo pallet. A mask transformation is applied to the cargo pallet in image B to obtain a 300×200 mask. and corresponding pixel blocks Superimpose it onto the fork of image A, starting coordinates At a point within the overlay area, the original pixel value of image A is 110, the corresponding pixel value of the cargo pallet is 190, and the mask value is 1. Therefore, the enhanced sample pixel value is... After overlaying, enhanced samples are generated, and the labels are updated to: forklift body. Forklift Goods pallets Then, select image A to generate an illumination-enhanced sample, and randomly select... , A pixel with an original pixel value of 150 is transformed into Finally, 2,000 real-labeled samples, 4,000 augmented samples, and 4,000 illumination-enhanced samples were mixed to form a small sample augmentation training set of 10,000 images.

[0054] Optionally, the step of performing temporal trajectory fitting on the multi-target detection results of consecutive frames to extract temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver limb posture changes, and distance between pedestrians and the fork area includes: smoothing the pixel coordinates of each target in consecutive frames using Kalman filtering to obtain the motion trajectory curve of each target; calculating the forklift speed and steering angle based on the inter-frame displacement and orientation angle changes of the forklift body target on the motion trajectory curve; calculating the fork lifting displacement amplitude based on the vertical coordinate changes of the fork target on the motion trajectory curve; extracting driver limb posture change features based on the inter-frame coordinate changes of the skeletal key points within the driver's human body contour; calculating the distance between pedestrians and the fork area based on the inter-frame changes of the coordinates of the pedestrian target and the boundary of the fork area; and combining the forklift speed, steering angle, fork lifting displacement amplitude, driver limb posture change features, and distance between pedestrians and the fork area features to form a temporal motion feature vector.

[0055] Specifically, this step transforms the output frame-by-frame, discrete multi-target detection results into continuous features capable of describing dynamic behavior through temporal analysis. Its core objective is to perform temporal trajectory fitting on the multi-target detection results across multiple consecutive frames, extract key dynamic information, and ultimately generate a structured temporal motion feature vector, providing a quantitative basis for subsequent violation judgment. The first step in the processing flow is to smooth the pixel coordinates of each target in consecutive frames using Kalman filtering. Since the output of the target detection network on consecutive frames may have slight jitter or momentary missed detections, directly using the original pixel coordinates to calculate motion parameters would introduce noise. Kalman filtering is an algorithm that uses the state equation of a linear system to optimally estimate the system state using system input and output observation data. Here, the detected target position in each frame is treated as a noisy observation, and combined with the target's motion model, its true position in the next frame is predicted, resulting in a smooth and continuous motion trajectory curve. For each target, its state vector... It can be defined as a combination of position and velocity, for example The filtering process consists of two stages: prediction and update. Prediction stage: Update phase: in, Represents the sequence number of the current frame. It is the first The final estimated state of the frame. It is the state transition matrix, which describes the state transition of the target from... Frame to The motion patterns of frames are usually set based on a uniform motion model. It is the first The observed values ​​of a frame are the pixel coordinates output by the object detection. It is the observation matrix, which maps the state vector to the observation space. It's the Kalman gain, which determines the level of confidence in observations based on the uncertainty between prediction and observation. Through this process, a smooth motion trajectory curve is generated for each continuously tracked target. After obtaining the smooth motion trajectory curve, various temporal features are extracted. First, based on the inter-frame displacement and orientation angle changes of the forklift vehicle target on the motion trajectory curve, the forklift's travel speed and steering angle are calculated. Forklift travel speed... It is obtained by calculating the displacement of its centroid between two consecutive frames and dividing by the time interval, and a calibration coefficient is needed to convert the pixel distance into the actual physical distance.

[0056] ;

[0057] in, and The forklift body target is at the 1st Frames and The smoothed centroid coordinates of the frame. It is the time interval between video frames, determined by the video frame rate. This is the conversion factor from pixel to physical distance, measured in meters per pixel. This factor is obtained through camera calibration, for example, by placing an object of known size in the scene. (Turning angle) This is obtained by calculating the angle between the displacement vectors at three consecutive moments, reflecting the change in the direction of travel. Secondly, based on the change in the vertical coordinate of the fork target on the motion trajectory curve, the lifting displacement amplitude of the fork is calculated. This value directly reflects the lifting and lowering action of the forks.

[0058] ;

[0059] in, and The target center of mass of the forklift is in the first position. Frames and The smoothed vertical coordinates of the frames. Driver limb pose change features are extracted based on the inter-frame coordinate changes of skeletal keypoints within the driver's body contour. After detecting the driver's body contour, a lightweight pose estimation model further locates skeletal keypoints such as the shoulder, elbow, and wrist. Driver limb pose change features. The displacement vectors of these key points across consecutive frames are used to determine whether the driver is making large, abnormal body movements. Based on the inter-frame changes in the coordinates of the pedestrian target and the forklift area boundary, the distance characteristics between the pedestrian and the forklift area are calculated. This feature is used to monitor whether pedestrians have entered hazardous work areas. The calculation method involves finding the shortest Euclidean distance between the pedestrian's bounding box and the forklift area's bounding box. Combining the extracted forklift speed, steering angle, forklift lifting displacement, driver's posture changes, and the distance between the pedestrian and the forklift area, these features are grouped into a vector to form a time-series meaningful final output that comprehensively describes the dynamics of the current work scenario: the time-series motion feature vector. .

[0060] For example, assuming the video frame rate is 25 frames per second, the inter-frame time interval... Seconds. The pixel-to-physical distance conversion coefficient is determined through camera calibration. Meters per pixel. The system processes three consecutive frames. The multi-target detection results are shown. After Kalman filtering and smoothing, the centroid coordinates of the forklift vehicle targets are obtained as follows: frame , No. frame , No. frame Calculate the first... Forklift travel speed in frame: pixel displacement is: Pixel. meters per second. Calculate the first... Frame rotation angle: 1st frame displacement vector . No. frame displacement vector . The radius is 36.87 degrees. Meanwhile, the smoothed centroid vertical coordinate of the fork target is at the [missing value]. Frame as In the Frame is Calculate the lifting displacement of the forks: Meters. In the first The frame detected the coordinates of the key point on the driver's right wrist as follows: , and in the Frame as The characteristics of driver's body posture changes. The right wrist displacement vector included is Furthermore, in the... The bounding box of the pedestrian target is in the frame. The bounding box of the forklift area is The two overlap horizontally, and the shortest distance is the lower edge of the pedestrian bounding box in the vertical direction. ) and the upper edge of the fork area ( The distance is: (pixel distance is:) Pixel. Meters. Finally, these calculation results are combined to generate the first... Temporal motion feature vector of a frame: This vector fully describes the forklift dynamics at the current moment.

[0061] Optionally, the rule matching reasoning unit retrieves the vectorized safety specification index in the local RAG knowledge base for semantic matching and outputs a violation determination result by combining static visual features and temporal motion feature vectors. This includes: converting the target category and contour size contained in the static visual features, and the motion parameters contained in the temporal motion feature vectors, into a query statement describing the current operation status; using the query statement to perform vector similarity retrieval in the vectorized safety specification index to recall multiple candidate safety specification clauses; reordering the recalled candidate safety specification clauses and selecting the one with the highest similarity as the matching clause; and logically comparing the constraints specified in the matching clause with the static visual features and temporal motion feature vectors to generate a violation determination result.

[0062] Specifically, in the preparation phase, two types of textual knowledge sources are first acquired: general forklift safety regulations and customized management regulations for specific locations within the park. To enable the computer to understand and retrieve these natural language rules, they need to be vectorized. This processing employs a pre-trained deep semantic understanding model, which can map any text fragment into a high-dimensional real-valued vector space. In this vector space, texts with similar semantics have corresponding vectors that are spatially close. Each safety regulation text is transformed into a fixed-dimensional vector, for example, 768-dimensional. All these text vectors together constitute the vectorized safety regulation index. This index is stored in a local RAG knowledge base. The local RAG knowledge base is a knowledge system that combines information retrieval and generation capabilities; here, it is specialized for "retrieval and matching," meaning it retrieves the most relevant rules based on real-time conditions rather than generating new text. The local deployment of the knowledge base ensures that the system can independently complete reasoning tasks even without a network connection. During the real-time operation monitoring phase, the rule matching and reasoning unit begins operation. First, two types of information are integrated: the target category and contour size extracted from the multi-target detection results as static visual features, and the temporal motion feature vector. To enable effective retrieval in the vectorized safety specification index, these numerical features need to be converted into semantic queries. This process is accomplished by converting the target category and contour size contained in the static visual features, and the motion parameters contained in the temporal motion feature vector, into a query statement describing the current operational state. This is a template-based generation process; for example, when a "pedestrian" target is detected, and the feature value of "pedestrian distance to forklift area" is 1.5 meters, the query statement "pedestrian is approaching the forklift area" can be generated. After generating the query statement, vector similarity retrieval is performed in the vectorized safety specification index using this query statement. The query statement is also converted into a query vector by the aforementioned deep semantic understanding model. The cosine similarity between this query vector and each safety specification vector in the knowledge base is calculated to measure their semantic relevance. The formula for calculating cosine similarity is:

[0063] ;

[0064] in, Represents the cosine similarity score, with a range of values. The closer the value is to 1, the more similar the semantics. It is a vector representation of the current job status query statement. It is a vector representation of a security specification text in the knowledge base. and These are the two vectors at the th... Dimensional components, This refers to the vector dimension. The system recalls multiple candidate safety specification clauses, typically those with similarity scores higher than a preset threshold. This threshold is based on a similarity score of 0.85 set for matching 1000 rules with the query. Next, the recalled candidate safety specification clauses are reordered, and the clause with the highest similarity is directly selected as the matching clause. The constraints specified in the matching clause are logically compared with static visual features and temporal motion feature vectors. Matching clauses usually contain explicit quantitative constraints, such as "driving speed must not exceed 5 km / h" or "cargo height must not exceed 2 meters." The rule matching inference unit parses these constraints and compares them with actual values ​​such as "forklift driving speed" in the temporal motion feature vector or the outline size of the "cargo pallet" in the static visual features. If the actual value violates the rule's constraints, the system generates a corresponding violation judgment result.

[0065] For example, suppose the local RAG knowledge base already stores two rules that have undergone text vectorization: Rule 1: "The forklift speed on the main road of the park shall not exceed 5 km / h.", and its vector is... Rule 2: "Personnel are strictly prohibited from entering the danger zone below the forks after they are raised; a safe distance of at least 2 meters must be maintained." Its vector is... At a certain moment, the system receives the following information: Static visual features: A "pedestrian" target has been detected, with an outline size of [missing information]. Pixel; "Forklift" target detected. Temporal motion feature vector: includes "forklift speed" of 2.1 m / s and "distance between pedestrian and forklift area" of 1.2 m. The system first converts these features into a query statement: "Pedestrian is close to forklift while forklift is in motion." This statement is then converted into a query vector. Perform vector similarity retrieval. Calculate... Cosine similarity with the rule vectors in the knowledge base: Assume the calculation result is: ; The system recalls rules with a similarity score higher than 0.85, i.e., rule 2. Since only one rule is recalled, it directly becomes the matching clause: "Personnel are strictly prohibited from entering the danger zone below the forks after lifting; maintain a safe distance of at least 2 meters." Finally, the rule matching inference unit performs a logical comparison. The constraint condition "safe distance" is parsed from the matching clause. "Meters". Then, the actual value is obtained from the temporal motion feature vector: "distance between pedestrian and forklift area". Meters. Logical comparison performed: This condition is not met. Because the actual situation does not meet safety regulations, the system ultimately outputs a violation judgment result, determining the current status as "pedestrian has entered the hazardous area for forklift operation," which is a violation.

[0066] Optionally, the constraints specified in the matching terms are logically compared with the static visual features and temporal motion feature vectors to generate violation judgment results, including: when the constraints of the matching terms determine that the driver is not wearing a seat belt properly, the forklift is carrying passengers illegally, personnel are standing on the fork pallet, the cargo is too high or too wide and obstructs the driver's view, a single fork is carrying a heavy load, the forks are raised or lowered during driving, speeding or driving against the flow of traffic in the factory area, or pedestrians enter the dangerous area of ​​fork operation, the corresponding violation type identifier and violation confidence level are output.

[0067] Specifically, the retrieved abstract safety rules are compared with the concrete and quantified visual evidence extracted from the video for a final logical judgment. The core of this process is to logically compare the constraints stipulated in the matching clauses with static visual features and temporal motion feature vectors. When the comparison result shows that the current operating state violates the constraints, a clear violation judgment result is generated. This logical comparison incorporates a composite judgment logic set for various specific violations. When a specific state occurs in the constraint judgment of the transmitted matching clauses, the system activates the corresponding judgment module, outputting the corresponding violation type identifier and violation confidence level. These states include: for the judgment of "driver not wearing seat belt properly," the system relies on static visual features. The logical comparison checks whether the "seat belt" target is not detected in the expected chest area while detecting the "driver's human silhouette." If this occurs in multiple consecutive frames, it is judged as a violation. For the judgment of "forklift illegally carrying passengers," the system analyzes the target count and spatial relationship in the static visual features. The system uses logical comparison to calculate the number of "pedestrian" or "driver's human silhouette" targets appearing in the non-driver's seat area within the "forklift body" bounding box. A violation is determined if the number is greater than 0. For the determination of "personnel standing on forks and pallets," the system also analyzes static visual features. Logical comparison calculates the overlap between the bounding boxes of "pedestrian" or "driver's human silhouette" and the bounding boxes of "forks" or "cargo pallets." A violation is determined if the intersection-union ratio of the two bounding boxes exceeds an empirically set threshold. For the determination of "cargo exceeding height or width and obstructing the driver's view," the system combines static visual features with preset parameters. Logical comparison first compares the outline dimensions of the "cargo pallet" with the height and width limits in safety regulations. Simultaneously, it determines whether the bounding box of the "cargo pallet" obstructs the bounding box of the "driver's human silhouette" in the vertical or horizontal direction. A violation is determined if either condition is met. For the determination of "single fork carrying a heavy load," the system analyzes the relative positions of the "cargo pallet" and "forks" in static visual features. The logical comparison checks whether the centroid of the "cargo pallet" deviates significantly from the centerline of the overall "fork" area. If the deviation exceeds half the width of the forks, it is considered a violation. For the determination of "raising and lowering the forks during travel," the system combines dynamic parameters from the temporal motion feature vector. The logical comparison checks whether the "forklift speed" is greater than the threshold indicating the vehicle is in motion, and whether the "fork lifting displacement amplitude" is also greater than the threshold indicating intentional raising and lowering. If both conditions are met, it is considered a violation. For the determination of "speeding and driving against traffic within the factory area," the system relies on the temporal motion feature vector. The logical comparison compares the "forklift speed" with the speed limit specified in the matching terms. For driving against traffic, the system needs to pre-load map information of the park's road directions and judge by comparing whether the forklift's current position and travel direction vector are consistent with the map's specified direction.For determining whether a pedestrian has entered the hazardous area for forklift operation, the system uses the distance feature from the temporal motion feature vector. Logical comparison compares the numerical value of the "distance between the pedestrian and the forklift area" with the safe distance specified in the matching clause. If the actual distance is less than the safe distance, a violation is determined. When outputting the violation determination result, the system also generates a violation confidence score. This value quantifies the reliability of the judgment result. Its calculation integrates the category confidence labels of all relevant detection targets that constitute the violation judgment, as well as physical parameters describing the degree of violation.

[0068] ;

[0069] in, This is the final confidence level for the violation, with a value range of [value missing]. . It is the first person involved in constituting this violation. The category confidence label for each target, such as "personnel standing fork pallet operation". It is the pedestrian's confidence level. It is the confidence level of the forks. It is the total number of targets involved in the judgment. It is a vector of key physical parameters that describes the degree of violation, such as crossover ratio, overspeed value, and intrusion distance. It is a normalization function that maps physical parameters to a range of 0 to 1. The higher the degree of violation, the closer the function value is to 1. The design of this function ensures the consistency of dimensions.

[0070] For example, suppose the system is determining the action of "raising or lowering the forks while the forklift is in motion." The obtained matching clause is: "Raising or lowering the forks is strictly prohibited while the forklift is in motion." The constraint condition parsed by the rule matching inference unit is: when the "forklift travel speed..." At a speed of meters per second, the "fork lifting displacement amplitude" must be The system extracts the "forklift travel speed" of 2.0 m / s and the "fork lifting displacement amplitude" of 0.3 m from the temporal motion feature vector of the current frame. Simultaneously, in the basic target detection results constituting these two features, the category confidence label for "forklift body" is 0.98, and the category confidence label for "forks" is 0.95. The logical comparison process is as follows: Check the travel status: The current "forklift travel speed" is 2.0 m / s, which is greater than the threshold of 0.5 m / s, therefore the forklift is in a traveling state. Check the lifting status: The current "fork lifting displacement amplitude" is 0.3 m, which is greater than the threshold of 0.05 m, therefore the forks are performing a lifting operation. Both conditions are met simultaneously, and the logical comparison result is a violation of the constraints. The system then generates a violation judgment result and calculates the violation confidence. In this scenario, the targets involved in the judgment are the "forklift body" and the "forks". The key physical parameters are velocity and displacement amplitude, which are assumed to be... for simplicity, here... . , The confidence level of the forklift body. The system outputs the violation judgment result as follows: Violation type identifier: "Lifting forks while driving", violation confidence level: 0.931.

[0071] Optionally, based on the violation determination results, the violation risk level is determined, a graded alarm signal corresponding to the violation risk level is generated, key frame images of the violation and short video streams of the corresponding time period are captured, and metadata is added to generate an encrypted evidence collection ledger record, including: classifying the violation risk level into three levels: mild, moderate, and severe based on the violation determination results; when a severe violation is determined, a live audio-visual alarm command is generated simultaneously, a pop-up reminder command is pushed to the forklift vehicle terminal, an alarm message is sent to the safety management backend, and a handling work order is automatically generated; when a moderate violation is determined, a work order to be handled is generated and pushed to the area administrator account; when a mild violation is determined, a violation record mark is generated and stored in the backend according to category; key frame images and short video streams within the time period corresponding to the violation determination results are captured, and metadata including the park location number, forklift equipment identifier, violation occurrence timestamp, and violation behavior type identifier is added, encrypted, and stored to form an unalterable evidence collection ledger record.

[0072] Specifically, the goal of this step is to process the results in depth, achieving closed-loop management from incident discovery to response and handling. This includes assessing the severity of the violation, triggering corresponding alarms and handling procedures, and securing evidence for future tracing. The process first determines the violation risk level based on the violation assessment results. The system has a built-in risk assessment matrix that associates each type of violation with a basic risk score. These basic scores are pre-set based on industry safety standards and expert experience; for example, the basic score for "pedestrian entering the forklift operation danger zone" is higher than that for "driver not wearing a seatbelt properly." The final risk score... Determined by both the base score and the confidence level of the violation:

[0073] ;

[0074] in, It is the calculated final risk score. This is the basic risk score corresponding to the type of violation. Subsequently, based on the risk score, the violation risk level is divided into three levels: mild, moderate, and severe. The thresholds for level division are based on a statistical analysis of the severity of 200 real safety incident cases; for example, a score between 0-30 is mild, 31-70 is moderate, and 71-100 is severe. After determining the violation risk level, a corresponding graded alarm signal is generated, and differentiated handling procedures are initiated. When a violation is determined to be severe, the situation is considered urgent and requires immediate intervention. At this time, an on-site audible and visual alarm command is generated simultaneously, issuing a warning directly on-site via connected hardware. Simultaneously, a pop-up reminder command is pushed to the forklift's onboard terminal, forcing the driver to receive a clear warning message. An alarm message is also sent to the safety management backend, and a handling work order is automatically generated, pushing detailed information about the event to the highest-authority safety management personnel, requiring immediate action. When a violation is determined to be moderate, although the risk is not as urgent as severe, timely follow-up is still necessary. The system generates a pending work order and pushes it to the administrator account of the area to which the forklift belongs, who is responsible for subsequent verification and processing. When a violation is determined to be minor, it is usually a matter of standardization and does not constitute a direct safety threat. The system generates a violation record marker, which does not generate interfering alarms on the front end, but is instead summarized and stored in the back end according to the violation type identifier for subsequent trend analysis and targeted training. To ensure that all violations are traceable, an evidence collection and solidification process is performed. Keyframe images and short video streams are captured within the time period corresponding to the violation determination result. The keyframe image is the frame image where the violation is most obvious. The short video stream captures a period of time before and after the violation, such as 5 seconds before and after, to provide complete contextual information. Metadata is attached to these captured images and videos. Metadata is a data packet containing structured information, including the park location number, forklift equipment identifier, violation timestamp, and violation type identifier. To ensure the integrity and non-repudiation of the evidence, the images and videos with attached metadata, as well as the metadata itself, are encrypted and stored. The encryption uses an asymmetric encryption algorithm to ensure that only authorized personnel can decrypt and view the data. After encryption, an immutable evidence record is created and stored in the system.

[0075] For example, suppose the system determines a violation event of "a pedestrian entering a dangerous area for forklift operation", and the violation confidence level is... The score is 0.95. First, determine the level of risk for the violation. Consult the risk assessment matrix to find the basic risk score for "pedestrian entering the hazardous area forklift operation". The score is 80. Calculate the final risk score: According to the preset threshold, a score of 76 constitutes a severe violation. Due to the severity of the violation, the highest level of response is immediately initiated. A command is sent to the IP speaker connected to the edge computing gateway to play a voice alarm: "Danger! Personnel approaching, please stop work immediately!" Simultaneously, a pop-up alert containing the text "Severe Violation: Pedestrian enters dangerous area" is sent to the forklift terminal numbered T-08. An alarm message containing all event details is pushed to the safety management backend, and a handling work order is automatically created and assigned to the safety director. The frame with the most severe violation is identified as the keyframe image and captured. Simultaneously, based on the timestamp of this frame, 5 seconds are captured forward and 5 seconds forward to generate a 10-second short video stream. Assume the event occurred under the camera at gate 3 of warehouse A, with the park location number "WH-A-03". The forklift involved is identified as "T-08". The timestamp of the violation is "15:30:22.125". The violation type is identified as "pedestrian trespassing into the hazardous area forklift operation." The captured keyframe images, short video streams, and the aforementioned metadata are packaged, encrypted using the RSA-2048 algorithm, and an encrypted file is generated. This file is named and stored in the system's evidence database, forming a complete and tamper-proof evidence log.

[0076] Optionally, the method is deployed on an edge computing gateway: the edge computing gateway has a built-in image preprocessing module, an AI detection and inference module, a time-series trajectory analysis module, a local RAG knowledge base, and a rule matching and inference module; when the network connection between the edge computing gateway and the cloud management platform is interrupted, the edge computing gateway continues to execute the steps and caches the generated hierarchical alarm signals and encrypted evidence collection ledger records, and automatically synchronizes the cached data to the cloud management platform in batches after the network connection is restored.

[0077] Specifically, this step defines the physical deployment and operating environment of the entire method, ensuring the system's real-time performance, reliability, and business continuity under extreme conditions. The entire processing flow is deployed and executed on an edge computing gateway. An edge computing gateway is a computing device deployed near the data source, possessing independent computing, storage, and networking capabilities. It can complete data processing locally, thereby reducing dependence on remote cloud servers and lowering latency. The internal architecture of this edge computing gateway is specially designed, with built-in functional modules forming an efficient and self-consistent processing pipeline. It includes an image preprocessing module responsible for performing layered dynamic brightness equalization, local occlusion repair, and fork motion distortion correction. Next is the AI ​​detection and inference module, which carries a lightweight multi-target detection network specifically for forklifts, performing target recognition on the preprocessed video frames. Subsequently, the temporal trajectory analysis module performs temporal fitting on the multi-frame detection results, extracting temporal motion feature vectors. Simultaneously, the edge computing gateway also incorporates a local RAG knowledge base and a rule matching and inference module. These two modules jointly implement the storage of vectorized security specifications and perform semantic matching and logical comparison based on real-time features to output violation judgment results. Ultimately, all judgment results and alarm generation are handled by the main control program on the gateway. The core advantage of this end-to-end edge deployment lies in its high robustness. When the network connection between the edge computing gateway and the cloud management platform is interrupted, whether due to network equipment failure or external network fluctuations, the gateway will not stop working. It will switch to offline working mode and continue executing the complete process. In this mode, all generated outputs, including tiered alarm signals and encrypted forensic ledger records, cannot be immediately uploaded to the cloud. To prevent data loss, the edge computing gateway caches this generated local data. The data is temporarily stored in the gateway's non-volatile storage media, such as solid-state drives, and a cache queue is maintained. When the network connection is restored, the edge computing gateway can automatically detect changes in network status, for example, through periodic network connectivity probes. Once it is confirmed that the connection with the cloud management platform has been restored, the gateway automatically triggers a data synchronization mechanism, synchronizing the data in the cache queue to the cloud management platform in batches. The synchronization process employs breakpoint resumption and data verification mechanisms to ensure the integrity and accuracy of the data during transmission. Only after the cloud management platform confirms successful reception will the edge computing gateway clear the corresponding data from the local cache queue. The total amount of data that needs to be synchronized is calculated as follows:

[0078] ;

[0079] in, It is the total size of the data to be synchronized. This is the number of records currently in the cache queue. Represents the first in the queue Each record contains tiered alarm signals and corresponding encrypted evidence collection ledger records. This is a function that calculates the size of a single record. This formula is used to assess the network bandwidth and time required for synchronization.

[0080] For example, suppose an edge computing gateway is deployed in warehouse 2 of a logistics park, responsible for monitoring forklift operations in that area. At 10:30 AM, due to a switch failure in the data center, the network connection between the edge computing gateway and the cloud management platform is interrupted. After the gateway's health monitoring service sends heartbeat packets to the cloud server IP address 202.108.22.5 three times consecutively at 5-second intervals without receiving a response, it switches the system status to offline mode at 10:30:15. At 10:35:20, the gateway's camera captures a forklift (device identification: F-11) moving while its forks are clearly raising and lowering. The gateway's built-in modules work as scheduled: the image preprocessing module enhances the video frames; the AI ​​detection and inference module identifies the forklift and forks; the time-series trajectory analysis module calculates the forklift's speed as 1.5 m / s and the fork raising / lowering displacement as 0.4 m; the rule matching and inference module searches the local RAG knowledge base, matches the rule "fork raising and lowering is prohibited while in motion," and determines it to be a violation. This violation was classified as a medium risk, and a pending work order and an encrypted forensic log containing keyframes and short videos were generated. Due to the network outage, the work order and log were stored in a local SSD cache queue, and a new record was added to the queue. At 10:45:00, the network fault was resolved and network connectivity was restored. The gateway's heartbeat service successfully received a response from the cloud server at 10:45:05. The gateway's synchronization service was activated and detected an unsynchronized record in the cache queue. The system immediately packages the recorded data and sends it to the cloud management platform's API interface via secure HTTPS. After receiving and verifying the data, the cloud platform sends a confirmation message to the gateway. Upon receiving the confirmation, the gateway will record the data. Removed from the local cache queue. At this point, the data generated during the offline period has been completely and securely synchronized to the cloud management platform, achieving seamless business continuity.

[0081] It should be noted that all equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. An AI-based method for identifying forklift violations, characterized in that, The method includes: Acquire continuous video frames of the storage operation area, and perform adaptive image preprocessing on the continuous video frames to obtain an enhanced video frame sequence; The enhanced video frame sequence is input into a lightweight multi-object detection network for forklifts that has been trained with few sample enhancements. Multi-scale feature fusion and multi-class target synchronous detection are performed, and the multi-object detection results of each video frame are output. The multi-object detection results include the pixel coordinates, contour size and category confidence labels of the forklift body, cargo pallet, seat belt, driver's human outline, forks and surrounding pedestrians. Temporal trajectory fitting is performed on the multi-target detection results of multiple consecutive frames to extract temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver limb posture change and distance between pedestrian and fork area, and generate temporal motion feature vector; The system acquires the text of forklift safety regulations and the text of customized management and control regulations for the park. It then performs text vectorization processing on the text of forklift safety regulations and the text of customized management and control regulations for the park to generate a vectorized safety regulation index and store it in the local RAG knowledge base. It extracts the target category and contour size from the multi-target detection results as static visual features. It inputs the static visual features and the temporal motion feature vector into the rule matching inference unit. The rule matching inference unit retrieves the vectorized safety regulation index in the local RAG knowledge base, performs semantic matching, and combines the static visual features and the temporal motion feature vector to output the violation judgment result. Based on the violation determination results, the violation risk level is determined, a graded alarm signal corresponding to the violation risk level is generated, key frame images of the violation and short video streams of the corresponding time period are captured, and metadata is added to generate encrypted evidence collection ledger records.

2. The AI-based method for identifying forklift violations according to claim 1, characterized in that, The step of acquiring continuous video frames of the warehousing operation area and performing adaptive image preprocessing on the continuous video frames includes: Layered dynamic brightness equalization is performed on the continuous video frames, and the brightness of the image is automatically adjusted according to the pixel grayscale distribution of a single frame to obtain a brightness correction frame. The brightness correction frame is partially occluded. The occluded pixel area is detected and the occluded area is filled with pixels using the spatiotemporal information of adjacent frames to obtain the occlusion repair frame. The occlusion repair frames are subjected to fork motion distortion correction, and geometric transformation correction is performed according to the degree of distortion of the fork edge contour to generate an enhanced video frame sequence.

3. The AI-based method for identifying forklift violations according to claim 1, characterized in that, The lightweight multi-object detection network for forklifts, trained with few-shot augmentation, is a reconstruction based on the YOLO backbone network, including: A first cross-scale feature fusion branch is added to the feature fusion part of the network to fuse multi-level feature maps of large forklift vehicle body targets and small seat belt targets; A second cross-scale feature fusion branch is added to fuse multi-level feature maps of medium-sized cargo pallet targets and slender fork targets; The feature maps output by the first and second cross-scale feature fusion branches are concatenated with the original feature map and input together into the detection head to output multi-target detection results.

4. The AI-based method for identifying forklift violations according to claim 1, characterized in that, The training sample set of the forklift-specific lightweight multi-object detection network includes: Collect real sample images of forklift operations in logistics and warehousing parks, and annotate the forklift body, cargo pallet, seat belt, driver's human outline, forks and pedestrian targets with categories and bounding boxes to obtain real annotated samples; Mask transformation and dynamic cargo overlay are applied to the real labeled samples to generate enhanced samples under cargo occlusion conditions; By combining illumination simulation to generate illumination-enhanced samples under different illumination conditions, the real labeled samples, enhanced samples, and illumination-enhanced samples are mixed to form a small-sample enhanced training set, which is used to train a lightweight multi-object detection network for forklifts.

5. The AI-based method for identifying forklift violations according to claim 1, characterized in that, The temporal trajectory fitting of the multi-target detection results across multiple consecutive frames, extracting temporal features of forklift speed, steering angle, fork lifting displacement amplitude, driver's limb posture changes, and distance between pedestrians and the fork area, includes: Kalman filtering is applied to the pixel coordinates of each target in consecutive frames to smooth them, thus obtaining the motion trajectory curves of each target. Calculate the forklift's speed and steering angle based on the inter-frame displacement and azimuth angle changes of the forklift vehicle target on the motion trajectory curve; Calculate the lifting displacement amplitude of the forks based on the change in the vertical coordinate of the fork target on the motion trajectory curve; Based on the inter-frame coordinate changes of key points in the skeletal structure of the driver's human body, the driver's limb posture change features are extracted. Calculate the distance features between the pedestrian and forklift area based on the inter-frame changes in the coordinates of the pedestrian target and the boundary of the forklift area; By combining the forklift's travel speed, steering angle, fork lifting displacement amplitude, driver's body posture changes, and distance characteristics between pedestrians and the fork area, a temporal motion feature vector is formed.

6. The AI-based method for identifying forklift violations according to claim 1, characterized in that, The rule matching and reasoning unit retrieves the vectorized security specification index from the local RAG knowledge base, performs semantic matching, and combines static visual features and temporal motion feature vectors to output violation determination results, including: The target category and contour size contained in the static visual features, and the motion parameters contained in the temporal motion feature vector, are converted into a query statement describing the current operation status. The query statement is used to perform vector similarity retrieval in the vectorized security specification index to recall multiple candidate security specification clauses. The candidate safety specification clauses for the recall were reordered, and the clause with the highest similarity was selected as the matching clause. The constraints specified in the matching terms are logically compared with the static visual features and temporal motion feature vectors to generate a violation determination result.

7. The AI-based method for identifying forklift violations according to claim 6, characterized in that, The constraints specified in the matching terms are logically compared with the static visual features and temporal motion feature vectors to generate violation judgment results. These results include: when the constraints of the matching terms determine that the following situations occur: driver not wearing seat belt properly, forklift carrying passengers illegally, personnel standing on fork pallets, goods exceeding height or width and obstructing the driver's view, single fork carrying heavy loads, raising and lowering forks during driving, speeding or driving against traffic in the factory area, or pedestrians entering the dangerous area of ​​fork operation, the corresponding violation type identifier and violation confidence level are output.

8. The AI-based method for identifying forklift violations according to claim 1, characterized in that, Based on the violation determination results, the violation risk level is determined, a graded alarm signal corresponding to the violation risk level is generated, key frame images of the violation and short video streams of the corresponding time period are captured, and metadata is appended to generate an encrypted evidence collection ledger record, including: Based on the aforementioned violation determination results, the violation risk level is divided into three levels: mild, moderate, and severe. When a serious violation is determined, an on-site audible and visual alarm command is generated simultaneously, a pop-up reminder command is pushed to the forklift's on-board terminal, and an alarm message is sent to the safety management backend and a handling work order is automatically generated. When a violation is determined to be moderate, a pending work order is generated and pushed to the area administrator's account; When a violation is determined to be minor, a violation record is generated and stored in the background according to category; Keyframe images and short video streams within the time period corresponding to the violation determination results are captured, and metadata including park location number, forklift equipment identification, violation occurrence timestamp, and violation type identification is added, and then encrypted and stored to form an immutable evidence collection ledger record.

9. The AI-based method for identifying forklift violations according to claim 1, characterized in that, The method is deployed and executed on an edge computing gateway: The edge computing gateway has a built-in image preprocessing module, an AI detection and inference module, a time-series trajectory analysis module, a local RAG knowledge base, and a rule matching and inference module. When the network connection between the edge computing gateway and the cloud management platform is interrupted, the edge computing gateway continues to perform the steps and caches the generated hierarchical alarm signals and encrypted evidence collection records. After the network connection is restored, the cached data is automatically synchronized to the cloud management platform in batches.

10. An AI-based forklift violation recognition system, characterized in that, The system is used in the AI-based forklift violation recognition method as described in any one of claims 1-9, the system comprising: The video acquisition and preprocessing module is used to acquire continuous video frames of the warehousing operation area, and perform adaptive image preprocessing on the continuous video frames to obtain an enhanced video frame sequence. The multi-target synchronous detection module is used to input the enhanced video frame sequence into a lightweight multi-target detection network for forklifts that has been trained with few samples, perform multi-scale feature fusion and multi-class target synchronous detection, and output the multi-target detection results for each video frame. The multi-target detection results include the pixel coordinates, contour size and category confidence labels of the forklift body, cargo pallet, seat belt, driver's human outline, forks and surrounding pedestrians. The temporal trajectory fitting and feature extraction module is used to perform temporal trajectory fitting on the multi-target detection results of multiple consecutive frames, extract the temporal features of forklift driving speed, steering angle, fork lifting displacement amplitude, driver limb posture change and distance between pedestrian and fork area, and generate temporal motion feature vector. The rule matching and reasoning module is used to acquire forklift safety regulations text and park customized management regulations text, perform text vectorization processing on the forklift safety regulations text and park customized management regulations text, generate a vectorized safety regulations index and store it in the local RAG knowledge base, extract the target category and contour size from the multi-target detection results as static visual features, input the static visual features and the temporal motion feature vector into the rule matching and reasoning unit, the rule matching and reasoning unit retrieves the vectorized safety regulations index in the local RAG knowledge base, performs semantic matching, and outputs the violation judgment result by combining the static visual features and the temporal motion feature vector; The graded alarm and evidence collection record module is used to determine the violation risk level based on the violation judgment result, generate a graded alarm signal corresponding to the violation risk level, capture key frame images of violations and short video streams of corresponding time periods, and add metadata to generate encrypted evidence collection ledger records.

Images