Industrial vehicle safety control and traffic flow analysis method and system based on end-side large model

Abstract

The invention relates to an industrial vehicle safety control and traffic flow analysis method and system based on an end-side large model, and the method comprises the steps: obtaining a forklift comprehensive image sample, carrying out the marking of the forklift comprehensive image sample, and obtaining a marking data set which comprises a forklift data subset, a cargo data subset, and a fork personnel data subset; based on preset model training parameters, model training samples are obtained from the forklift data subset, the cargo data subset and the fork personnel data subset in a classified mode; based on a classified model training sample, performing classification training on the constructed end-side deep learning model until training is finished to obtain a forklift safety control-traffic flow analysis model; and obtaining a to-be-identified forklift comprehensive image, and analyzing the to-be-identified forklift comprehensive image based on the forklift safety control-traffic flow analysis model to obtain forklift safety control-traffic flow result data. The forklift can be safer in the operation process, and vehicle allocation is more timely and accurate.

Description

Technical Field

[0001] The present application relates to the field of industrial vehicle safety and control management, and in particular to an industrial vehicle safety and control and traffic flow analysis method and system based on a large end-side model. Background Art

[0002] Assisted driving systems (ADAS) have been widely used in the automotive field. ADAS collects data through various sensors installed on the vehicle (such as cameras, radars, etc.), monitors the vehicle's operating status and surrounding environment, and provides early warning or auxiliary control to improve driving safety and efficiency.

[0003] With the gradual development of ADAS in the automotive field, its application in industrial vehicles, especially forklifts, is also gradually emerging. Through cameras, lidar, ultrasonic sensors, etc., the forklift's surrounding environment can be monitored in real time, and warnings can be issued when pedestrians approach. Sensors can also monitor the weight and distribution of cargo in real time and alarm in the event of overload. On the one hand, warnings issued when pedestrians approach the vehicle are relatively one-sided in detecting the safety of people near the vehicle, and detecting cargo transported by forklifts by weight is not safe enough. On the other hand, the environment of different storage areas varies. When forklifts operate in different environments, the timeliness and accuracy of manual traffic statistics are low, making it difficult for managers to dynamically allocate vehicle resources in a timely manner according to actual conditions.

[0004] In summary, existing ADAS has obvious shortcomings in forklift safety detection and dynamic vehicle deployment in different working environments. There is an urgent need for a safer technology that can more timely and accurately deploy vehicles to meet the operational needs of forklifts. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the embodiments of the present application provide an industrial vehicle safety control and traffic flow analysis method and system based on a large end-side model, aiming to make forklifts safer in use and more timely and accurate in deployment.

[0006] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] A method for industrial vehicle safety control and traffic flow analysis based on a large end-to-end model, including:

[0008] Obtaining forklift comprehensive image samples and annotating the forklift comprehensive image samples to obtain an annotated data set, wherein the annotated data set includes a forklift data subset, a cargo data subset, and a forklift operator data subset;

[0009] Based on preset model training parameters, classify and obtain model training samples from the forklift data subset, the cargo data subset, and the forklift operator data subset;

[0010] Based on the classified model training samples, the constructed end-side deep learning model is classified and trained until the training is completed to obtain a forklift safety control and traffic flow analysis model;

[0011] A comprehensive image of a forklift to be identified is obtained, and the comprehensive image of the forklift to be identified is analyzed based on the forklift safety control-traffic flow analysis model to obtain forklift safety control-traffic flow result data.

[0012] Accordingly, the present application also provides an industrial vehicle safety control and traffic flow analysis system based on a large end-to-end model, including:

[0013] a sample data set acquisition module, configured to acquire forklift comprehensive image samples and annotate the forklift comprehensive image samples to obtain an annotated data set, wherein the annotated data set includes a forklift data subset, a cargo data subset, and a forklift operator data subset;

[0014] A sample acquisition module, configured to classify and acquire model training samples from the forklift data subset, the cargo data subset, and the forklift operator data subset based on preset model training parameters;

[0015] A model training module is used to perform classification training on the constructed end-side deep learning model based on the classified model training samples until the training is completed to obtain a forklift safety control and traffic flow analysis model;

[0016] The image analysis module is used to obtain a comprehensive image of the forklift to be identified, and analyze the comprehensive image of the forklift to be identified based on the forklift safety and control-traffic flow analysis model to obtain forklift safety and control-traffic flow result data.

[0017] The beneficial effects of this application are:

[0018] 1. This application obtains model training samples from a subset of forklift personnel data and can analyze the personnel on the forks. When it is detected that there is a person on the forklift forks, the security control-traffic analysis model can be used to timely analyze the image to facilitate timely alarm. Compared with the traditional alarm that can only detect whether pedestrians are close to the forklift and issue an alarm, this application can also issue an alarm if there is a person standing on the fork, which can further improve the safety of forklift operations.

[0019] 2. This application obtains model training samples from a subset of cargo data and can analyze the cargo on the forks. When cargo falls from the forks, the security control-traffic analysis model can promptly analyze the image to facilitate timely alarm. While recording the status of the cargo, it can also promptly remind nearby personnel to prevent secondary damage to the cargo as much as possible.

[0020] 3. This application obtains model training samples from a subset of forklift data to analyze the number and driving trajectories of forklifts. By timely analyzing images through the security control-traffic flow analysis model, it can promptly determine the vehicle allocation during the forklift operation process, making it convenient for relevant personnel to adjust the deployment of forklifts in a timely manner based on the allocation of forklifts and the actual conditions during driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of a process flow of an industrial vehicle safety control and traffic flow analysis method based on a large end-side model provided in an embodiment of the present application;

[0023] Figure 2 A structural diagram of an industrial vehicle safety control and traffic flow analysis system based on a terminal-side large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In industrial warehouse environments, forklifts are the primary means of transportation, and their operational safety and efficiency are crucial. Advanced Driver Assistance Systems (ADAS) have been widely used in the automotive sector, and their application in industrial vehicles, particularly forklifts, is also gradually gaining momentum. ADAS collects data from various sensors installed on the vehicle (such as cameras and radar), monitors the vehicle's operating status and surrounding environment, and provides early warnings or auxiliary control to improve driving safety and efficiency. At the same time, the development of large-scale edge model technology has opened up new possibilities for expanding ADAS functionality for industrial vehicles. These edge-based models enable local data analysis and processing, eliminating the need to transmit large amounts of data to the cloud. This reduces data transmission latency and privacy risks, and allows for customized model training and optimization based on specific scenarios.

[0025] However, during actual operations, operators may step on the forks due to insufficient safety awareness or negligence. Traditional forklift ADAS systems lack effective monitoring for this dangerous behavior, preventing timely detection and warning, which can easily lead to accidents. Furthermore, traditional ADAS systems struggle to monitor dropped cargo during transportation in real time. Dropped cargo can not only damage the cargo but also pose a safety threat to surrounding personnel and equipment. Existing systems are unable to issue and record timely warnings, hindering subsequent tracing and analysis of the cause of the accident.

[0026] In terms of forklift traffic management, when forklifts are operating within the factory, most warehouse and logistics sites currently rely on manual statistics or simple sensing equipment to obtain forklift traffic information. This method has low data accuracy and poor real-time performance, making it impossible to comprehensively and accurately analyze forklift traffic conditions in different time periods and areas. Due to the lack of accurate traffic analysis data, warehouse and logistics managers find it difficult to dynamically adjust work shifts and resource allocation based on actual conditions. During peak operating periods, forklift congestion often occurs, resulting in low logistics efficiency, low equipment utilization, and increased operating costs. In terms of warehouse management, inventory currently relies mainly on manual counting of goods one by one, which is time-consuming and labor-intensive, especially in large warehouses. Inventory work requires a large amount of manpower and time costs.

[0027] In response to the above-mentioned problems, an embodiment of the present application provides an industrial vehicle safety control and traffic flow analysis method based on a large end-side model, which specifically includes: S100, obtaining a forklift comprehensive image sample, and labeling the forklift comprehensive image sample to obtain a labeled data set, which includes a forklift data subset, a cargo data subset, and a forklift personnel data subset.

[0028] By acquiring comprehensive forklift image samples, we can obtain images of the forklift itself, images of the forklift forks, images of the cargo on the forks, and images of the person. Then, by annotating these images, we can obtain different types of images and related information about the person, cargo, and forklift, such as person location, person posture, cargo status, and forklift speed and trajectory, thus generating a labeled dataset.

[0029] The labeled dataset includes a forklift data subset, a cargo data subset, and a forklift personnel data subset. The forklift data subset includes images of forklifts and related information, the cargo data subset includes images of cargo and related information, and the forklift personnel data subset includes images of personnel on the forks and related information. These different types of data can be used to classify images, which facilitates subsequent model training.

[0030] Furthermore, in one embodiment, obtaining a comprehensive image sample of a forklift includes: based on an image acquisition device at a preset position, acquiring an image of the forklift to obtain a comprehensive image sample of the forklift; the image acquisition device at the preset position includes a first image acquisition device located on the forklift, and a second image acquisition device located at a hub position of the forklift route, the first image acquisition device being used to acquire an image of the forklift's forks, and the second image acquisition device being used to acquire an image of the forklift.

[0031] Specifically, the first image acquisition device can be a camera installed on a forklift. There is at least one camera, which can be installed near the fork or in front of the forklift cab. The camera installed on the forklift can capture the position of the fork and obtain relevant images of the fork, such as images of goods placed on the fork and images of people standing on the fork. Of course, these cameras can also capture images around the forklift, such as the road conditions in front of the forklift and the conditions around the forklift, so as to facilitate whether there are obstacles or pedestrians approaching around the forklift.

[0032] The second image acquisition device can be a camera installed at a hub location on the forklift's route. This camera, installed at a hub location along the factory route where the forklift operates, can capture images of the forklift as it travels along the road. A hub location is a location that the forklift must pass through during operation. By capturing images of the forklift, relevant information such as the forklift's route, location, and speed can be determined, facilitating subsequent training.

[0033] Furthermore, in one embodiment, forklift comprehensive image samples are labeled to obtain a labeled data set, including: determining personnel information, cargo information, and forklift information in the forklift comprehensive image samples; and labeling the forklift comprehensive image samples based on the personnel information, cargo information, and forklift information to obtain a labeled data set.

[0034] The forklift comprehensive image sample includes a forklift comprehensive image sample of personnel and cargo captured by the first image acquisition device, and forklift information captured by the second image acquisition device. The forklift comprehensive image sample of personnel and cargo captured by the first image acquisition device can be identified using an image recognition algorithm. That is, the personnel and cargo in the forklift comprehensive image sample are identified using a preset image recognition algorithm to obtain personnel information and cargo information. In this application, the steps for identifying using the preset image recognition algorithm are as follows:

[0035] First, we correlate the forklift's fork angle with the object's posture. When a forklift is operating, the forks will tilt at different angles, and the posture of people and cargo will differ at different fork angles. Let the fork's current tilt angle be θ, which can be obtained through the forklift's sensor. In the image, calculate the object's center of gravity offset, O, using the following formula: Among them, (x i ,y i ) is the coordinate of the i-th pixel of the object, is the average x-coordinate of all pixels of the object, w i is the weight of the pixel (which can be determined based on features such as the grayscale or color of the pixel value).

[0036] Next, define a posture correlation factor A: A = k·θ·O, where k is an adjustment coefficient. For a person standing on the fork, when the fork tilts, the person will adjust the body posture to maintain balance, resulting in a relatively large center of gravity offset and more related to the fork angle; while when the fork tilts, the cargo usually just tilts along with the fork, and the center of gravity offset is relatively regular. This application embodies the unique feature of the change in fork angle during forklift operation by combining the fork angle and the object's center of gravity offset, which is used to distinguish between people and cargo. Next, determine the adaptability of the fork spacing of the forklift. The fork spacing of the forklift is adjustable, and cargo and personnel of different sizes have different adaptability to the fork spacing. Let the fork spacing be d, the lateral coverage of the object on the fork be L, and define a spacing adaptation factor F: When cargo is on the forks, the size and shape of the cargo is relatively fixed, and the F value is usually within a relatively stable range. However, when a person stands on the forks, the F value fluctuates more due to different standing positions. This application utilizes the adjustable spacing between the forks of a forklift to assist in determining whether the object on the forks is a person or cargo.

[0037] Then, determine the object shaking characteristics when the forklift is in motion. The forklift will have motion states such as acceleration, deceleration, and turning during driving and operation. The shaking conditions of people and goods in these motion states are different. Suppose the acceleration change of the forklift in N consecutive frames is Δa, and the standard deviation of the pixel position change of the object in the corresponding frame is σ, and define a shaking correlation factor H: H = m·Δa·σ. Among them, m is the adjustment coefficient. When the forklift is moving, the person will reduce the shaking through his own balance adjustment ability, so when the acceleration of the forklift changes, the standard deviation of the person's position change is relatively small; while the goods are more likely to shake with the movement of the forklift, and the standard deviation of the position change is larger. This application combines the forklift's motion state and the object shaking characteristics, which can better judge people and goods.

[0038] Finally, based on the above three factors, this application defines a comprehensive recognition index R: R = γA + δF + ∈H, where γ, δ, ∈ are weight coefficients that can be adjusted according to actual conditions. Set a threshold T R , when R>T R When R≤T R , it is judged as goods.

[0039] The aforementioned recognition algorithm accurately analyzes the information of the person and cargo on the forks from the images captured by the first image acquisition device. For forklift information, existing image recognition algorithms can be used to determine whether a forklift is present in the images captured by the second image acquisition device. Once the person, cargo, and forklift are identified within the forklift comprehensive image sample, the forklift comprehensive image sample can be simply labeled, enabling simple classification of the sample and resulting in a labeled sample dataset, also known as a labeled dataset.

[0040] Through the above-mentioned method, the technical solution provided by this embodiment has the following technical advantages:

[0041] 1. Through image acquisition devices at different positions, different images of forklifts can be collected in a targeted manner, so that the images can be better classified and recognized, and the corresponding images can be obtained more conveniently for targeted model training.

[0042] 2. For the identification of people and goods, it can combine the actual situation of the forklift itself, such as the angle and spacing of the forks and the shaking characteristics of the objects in motion, to identify the people and goods on the forklift forks, which can better meet the accuracy of the forklift's identification of people and goods.

[0043] S200 , based on preset model training parameters, classify and obtain model training samples from the forklift data subset, the cargo data subset, and the forklift operator data subset.

[0044] Because training samples can be trained for different training models, the system presets different model training parameters. These parameters can be related to the model structure. For example, for a Transformer structure, if the model contains a Transformer module, such as a self-attention mechanism, it is a key indicator of a Transformer-based model. The self-attention mechanism allows the model to adaptively focus on information at different positions when processing sequence data or image data, and has strong capabilities for modeling long sequence dependencies.

[0045] For recurrent layers or time series processing modules, network models that incorporate spatiotemporal features, particularly for tasks involving time series data such as transportation flow analysis, often include recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) and gated recurrent units (GRU). These recurrent layers can handle long-term dependencies in time series and capture trends in data over time. The presence of modules specifically designed for time series processing, such as temporal convolutional networks (TCNs), also indicates a focus on extracting temporal features.

[0046] Furthermore, in some embodiments, based on preset model training parameters, model training samples are classified and obtained from the forklift data subset, the cargo data subset, and the forklift personnel data subset, including: determining the data category of the forklift data subset, the cargo data subset, and the forklift personnel data subset based on the annotation information of the labeled data set; determining the model category of the end-side deep learning model based on the preset model training parameters; and matching the data category with the model category, and obtaining model training samples from the forklift data subset, the cargo data subset, and the forklift personnel data subset.

[0047] By determining the model training parameters, model training samples can be obtained by classification from the forklift data subset, cargo data subset and forklift personnel data subset. For example, when annotating the forklift comprehensive image sample, the training type to which the image sample belongs can also be annotated, that is, the annotation information includes the training type to which the image sample belongs. For example, the cargo data subset and the personnel data subset are annotated as training subsets of the Transformer structure, and the forklift data subset is annotated as a training subset of the recurrent layer or time series processing module, so that the data category of the forklift data subset, cargo data subset and forklift personnel data subset can be determined. After determining the model category of the model to be trained, the forklift data subset, cargo data subset and forklift personnel data subset can be associated with the model of the corresponding category respectively, so that during the model training process, the required samples can be obtained from the corresponding subset to achieve classification and obtain model training samples.

[0048] In this application, the following methods can be used to obtain model training samples:

[0049] Let the forklift data subset be D f , the cargo data subset is D g , the forklift personnel data subset is D p Define the labeling function L(·), which accepts the data subset and the related feature vector as input and outputs the labeling information. For the forklift data subset, consider the forklift’s driving speed v f , operation time t f , historical trajectory fluctuation coefficient σ f (reflecting the stability of the forklift's driving trajectory) and other features, forming the feature vector F f =[v f ,t f ,σ f ]. Label information L(D f )for:

[0050] Where C1 is the training type set of the forklift data subset (such as the recurrent layer or time series processing module related categories), S1 (Ff ,c) is a similarity calculation function, which measures the feature vector F through nonlinear transformation f The degree of match with training type c. For example: where α i is the weight coefficient, φ i (F f ,c) is the eigenvector F f The correlation calculation function between the i-th feature and the training type c is as follows (β1 is the adjustment coefficient, μ c1 is the mean speed corresponding to training type c).

[0051] For the cargo data subset, consider the weight w of the cargo g , volume V g , shape complexity index τ g (calculated by Fourier descriptor of cargo contour and other methods) and other features to form the feature vector F g =[w g ,V g ,τ g ]. Label information L(D g )for: Where C2 is the training type set of the cargo data subset (such as Transformer structure related categories), S2 (F g ,c) is the similarity calculation function, for example: where γ i is the weight coefficient, ψ i (F g ,c) is the eigenvector F g The correlation calculation function between the i-th feature and the training type c is as follows (ω c2 and ω c2,k is the shape complexity weight associated with training type c).

[0052] For the forklift personnel data subset, consider the personnel's movement amplitude a p , action frequency f p , relative position stability with the fork δ p Equal features, forming the feature vector F p =[a p ,f p ,δ p ]. Label information L(D p )for: Among them, S3(F p ,c) is calculated in the same way as S2(F g ,c) is the same, and the matching degree between the feature vector and the training type is also measured through a nonlinear function.

[0053] Assume that the model set is M={M1,M2,…,M n}, where M1 is a model based on a recurrent layer or a time series processing module, M2 is a model based on a Transformer structure, etc. Define an association function A(·) that associates a data subset with an appropriate model.

[0054] Among them, M C1 is a set of models related to the training type of the forklift data subset (i.e., a set of models based on recurrent layers or time series processing modules), E(D f ,M i ) is an evaluation function used to measure the data subset D f With Model M i For example: where x j ∈D f is a sample in the forklift data subset, x j,k is the sample x j The kth feature (corresponding to F f features in), T(x j ) is the sample x j In the ideal output (which can be obtained by expert annotation or other reliable methods), θ k is the feature weight coefficient.

[0055] For the association between the cargo data subset and the forklift personnel data subset and the model, Among them, M C2 is a set of models related to the training type of the cargo and forklift personnel data subset (i.e., a set of models based on the Transformer structure), E(D g ,M i ) and E(D p ,M i ) is calculated in the same way as E(D f ,M i ) but will be adjusted accordingly based on the characteristics of the goods and personnel data.

[0056] Through the above technical solution, this application has the following technical advantages:

[0057] 1. Existing data labeling and model association formulas are usually relatively simple, and are only classified based on the surface attributes of the data. For example, they are only labeled according to the type of data (forklift, cargo, personnel), and then randomly or with simple rules to associate the model. However, this application fully considers the intrinsic characteristics of forklift, cargo and personnel data, and performs labeling and association through nonlinear functions and multi-dimensional feature analysis. In the labeling process, a variety of features related to forklift operations (such as forklift driving speed, operation time, etc.) are introduced, and the training type is determined by the similarity calculation function; when associating the model, the degree of adaptation of the data and the model is measured by the evaluation function, and the model's processing ability of the data features and the output of the data under ideal conditions are comprehensively considered.

[0058] 2. The formula design of this application can more accurately determine the training type and associate the appropriate model according to the actual characteristics and needs of the data. For forklift data, considering its time series related characteristics (such as driving speed, historical trajectory fluctuation coefficient), select the model of the loop layer or time series processing module for training, which can better learn the dynamic behavior pattern of the forklift; for cargo and fork personnel data, considering its shape, movement and other characteristics, select the model of the Transformer structure, which can better handle the long-distance dependencies and complex features in the data. Compared with traditional methods, this application can improve the efficiency and accuracy of model training, make the trained model more in line with the actual needs of forklift operation scenarios, and perform better in tasks such as fork station person detection, cargo drop detection, and forklift flow scheduling detection.

[0059] Furthermore, in some embodiments, model training samples are obtained by classification from the forklift data subset, the cargo data subset, and the forklift personnel data subset, including: identifying the forklift data subset, the cargo data subset, and the forklift personnel data subset to obtain interference data, the interference data including invalid data, erroneous data, and duplicate data; screening out the interference data and classifying to obtain model training samples.

[0060] Specifically, in the embodiment of the present application, the interference data can be identified in the following ways:

[0061] First, for invalid data identification, in this application, for the forklift data subset D f , let the sample x f ∈D f , define the effective range of forklift operation time as [t min ,t max ], the effective range of driving speed is [v min ,v max Invalid data judgment function I f (x f )for: where x f[t] represents the sample x f The operation time characteristics in x f [v] represents the driving speed characteristic.

[0062] For the cargo data subset D g , let the sample x g ∈D g , define the effective range of cargo weight as [w min ,w max ], the effective volume range is [V min ,V max Invalid data judgment function I g (x g )for: where x g [w] represents sample x g The weight feature in x g [V] indicates volume feature.

[0063] For the forklift personnel data subset D p , let the sample x p ∈D p , define the effective range of personnel movement range as [a min ,a max ], and the effective range of the relative position with the fork is [δ min ,δ max Invalid data judgment function I p (x p )for: where x p [a] represents sample x p The motion amplitude characteristics in x p [δ] represents the relative position characteristic to the fork.

[0064] For error data identification, in this application, for the forklift data subset, the rationality of the forklift's trajectory is considered. Let the sample x f The trajectory point sequence in is {p1,p2,…,p n}, by calculating the distance between adjacent trajectory points (p i [x] and p i [y] represents the trajectory point p i The average moving speed is defined as (t total is the total time). Error data judgment function E f (x f )for: where d thresh is the distance threshold, which is used to determine whether the trajectory point has abnormal jumps.

[0065] For the cargo data subset, consider the consistency of cargo shape. Let the sample x g The shape feature vector in is S g , by calculating the rate of change of shape features (S g-1 is the shape feature vector of the previous sample). Error data judgment function E g (x g )for: where r thresh is the shape change rate threshold.

[0066] For the identification of repeated data, in this application, for the forklift personnel data subset, the continuity of personnel actions is considered. Let the sample x p The action sequence in is {a1,a2,…,a m}, by calculating the difference between actions Error data judgment function E p (x p )for: Among them D thresh is the action difference threshold.

[0067] For the forklift data subset, let the sample x f and y f , define the duplicate data judgment function R f (x f ,y f )for: where x f [i] and y f [i] represents the sample x f and y f The i-th feature of , k is the number of features, and ∈ is the duplicate judgment threshold. Similarly, for the cargo data subset and the fork personnel data subset, the duplicate data judgment function R g (x g ,y g ) and R p (x p ,y p ) have the same form, except that the features and thresholds are set according to the characteristics of the respective data.

[0068] For interference data identification, in this application, for the forklift data subset, the subset D′ after the interference data is filtered out is f for: For the cargo data subset, the subset D′ after the interference data is filtered out g for: For the forklift personnel data subset, the subset D′ after the interference data is filtered out p for:

[0069] By adopting the above-mentioned method, the technical solution provided by this embodiment has the following technical effects:

[0070] 1. Traditional interference data identification and screening methods often use simple threshold judgments or comparisons based on a small number of features. For example, invalid data is judged only based on a single numerical range, and duplicate data is simply compared with some key features. However, this application comprehensively considers features and business logic in multiple dimensions. In invalid data identification, multi-dimensional valid range judgments are set according to the characteristics of different data subsets (forklifts, goods, personnel); in erroneous data identification, business logic such as the rationality of forklift trajectories, the consistency of cargo shapes, and the continuity of personnel movements are considered; in duplicate data identification, a comprehensive comparison of multiple features is used to determine whether the data is duplicated.

[0071] 2. The design of this application can more accurately identify and filter out interference data. In the forklift operation scenario, the driving trajectory of the forklift is affected by many factors. It is difficult for traditional methods to accurately judge trajectory abnormalities. This application judges erroneous data by calculating multiple indicators such as the distance between trajectory points and the average moving speed. It can effectively eliminate erroneous trajectory data caused by equipment failure, signal interference and other reasons. For cargo data, this application judges erroneous data by the shape change rate, and can find erroneous shape data caused by data collection errors or abnormal changes in cargo. In terms of duplicate data recognition, this application conducts a comprehensive comparison of multiple features to avoid misjudgment caused by similar features but different overall data. Through these improvements, the data quality can be greatly improved, thereby improving the effect and reliability of model training, so that the trained model performs better in forklift operation-related tasks.

[0072] Furthermore, in some embodiments, before performing classification training on the end-side deep learning model, the method further includes: performing data enhancement on the model training samples, where the data enhancement includes angle transformation, size transformation, and area transformation.

[0073] Specifically, data angle transformation involves rotation, which involves rotating the captured image data at specific angles (e.g., ±15°, ±30°, ±45°, etc.) to simulate images captured by a camera at different shooting angles. For example, a horizontal image of the scene in front of a forklift can be rotated to produce a similar image from an oblique angle. This allows the model to accurately identify cargo presented at various angles in real-world applications, enhancing its adaptability to angle changes.

[0074] The angle change includes the following implementation methods: introducing an adaptive rotation parameter α, which dynamically adjusts the rotation angle according to the content characteristics of the image. Assume that through edge detection or feature point extraction of the image, a feature value F related to the object structure in the image is obtained, and then define α = f(F), where f is a custom function, for example k is a constant, F min and F max are the minimum and maximum values ​​of the eigenvalue F, respectively.

[0075] Then, the image rotation formula can be expressed as:

[0076] In this way, there are the following technical benefits:

[0077] The rotation angle is no longer fixed, but is adaptively adjusted based on the image's inherent characteristics, more accurately simulating the effect of viewing the image from different oblique angles. For example, in an image of a scene in front of a forklift, if the detected edge features of the cargo are distinct and have a certain directionality, α will be adjusted based on this feature value, making the rotated image more similar to the image actually viewed from a specific oblique angle, enhancing the model's adaptability to angle changes.

[0078] Resizing involves scaling, which involves applying different zoom ratios (e.g., 0.8x, 1.2x, 1.5x, etc.) to an image. This includes both zooming out to simulate the appearance of an object at a greater distance, and zooming in to simulate close-up observation of details. For example, scaling an image of a person on a forklift allows the model to learn the different characteristics of the human body. This allows accurate identification of the person, regardless of whether they occupy a larger or smaller portion of the camera's field of view.

[0079] Region transformations include cropping operations, which involve cropping images according to different cropping regions and sizes to highlight key parts of the image or simulate partial occlusion of objects. For example, cropping out the local area of ​​an image containing a person allows the model to focus on learning key features such as human posture. Or, simulating a scene of falling goods by cropping the image to create a similar visual effect can train the model to accurately identify goods even in the presence of real occlusion. These cropping operations further enrich the diversity of the dataset and improve the robustness of the model.

[0080] S300: Based on the classified model training samples, the constructed end-side deep learning model is classified and trained until the training is completed to obtain a forklift safety control and traffic flow analysis model.

[0081] In this embodiment, the model training method can be implemented in the following way:

[0082] Assume that the total sample set is S, which contains the forklift data sample S f , cargo data sample S g , Forklift personnel data sample S p , that is, S=S f ∪S g ∪S p The division ratio is α for the training set, β for the validation set, and γ for the test set, and α+β+γ=1.

[0083] Using the stratified sampling method, for the forklift data sample S f , the divided training set S f-train , validation set S f-val , test set S f-test satisfy: Where |·| represents the number of elements in the set. Similarly, we can get the set S after the cargo data sample and the forklift personnel data sample are divided g-train 、S g-val 、S g-test With S p-train 、S p-val 、S p-test .

[0084] Final training set Train = S f-train ∪S g-train ∪S p-train , validation set Val = S f-val ∪S g-val ∪S p-val , test set Test = S f-test ∪S g-test ∪S p-test .

[0085] Model for identifying people standing on the forklift and cargo falling: It uses a CNN and Transformer architecture. Let the input image be I, and it passes through n convolution layers (the convolution kernel is K i , i=1,2,…,n) and the pooling layer operation to obtain the feature map F=Pooling n (Convolution n (…(Convolution1(I,K1),K2)…,K n )).

[0086] The feature map F is flattened into a vector V and input into the Transformer module. The Transformer module contains m encoding layers, and the multi-head attention mechanism of each encoding layer is calculated as follows: Where Q, K, and V are query, key, and value vectors respectively, d kis the dimension of the key vector. After the Transformer module, the output vector O is obtained. cnn-trans .

[0087] Transport flow analysis model: a network model combining spatiotemporal features. Let the input spatiotemporal data of the forklift be X, which contains the position coordinates (x, y), time t, speed v and other features. Through the spatiotemporal convolution layer (the convolution kernel considers the time dimension and the space dimension, denoted as K st ) and recurrent layers (such as LSTM) process: h t ,c t =LSTM(X t ,h t-1 ,c t-1 ) where h t 、c t They are the hidden state and cell state of LSTM at time t, and the output vector O is finally obtained. st .

[0088] The model parameters are initialized using the Xavier initialization method. For the weight matrix W, the element initialization formula is: where n in is the number of input neurons, n out is the number of output neurons, and U represents uniform distribution.

[0089] For the classification task of identifying people standing on the fork and cargo falling, the cross entropy loss function is used. Let the true label be y (y is a one-hot encoding vector) and the probability distribution predicted by the model be p, then the loss function L ce for: For the regression task of transportation flow analysis, the mean square error loss function is used. Let the true value be y true , the model prediction value is y pred , then the loss function L mse for: Where N is the number of samples.

[0090] Model parameter update uses the Adam optimization algorithm, and the update formula of parameter θ is: where g t is the gradient at time t, β1 and β2 are the exponential decay rates of moment estimates, η is the learning rate, ∈ is a minimum constant to prevent the denominator from being zero, and m t 、v t is an intermediate variable.

[0091] Let the number of training rounds be epoch and the current round be e. According to the loss value L on the validation set val And Accuracy Acc val Adjust the learning rate η and data enhancement strength s. Learning rate adjustment: Use cosine annealing learning rate adjustment strategy: where η min ,η max are the minimum and maximum values ​​of the learning rate, respectively, and E is the total number of training rounds.

[0092] Data enhancement intensity adjustment: set the validation set loss change rate When ΔL val When >δ1 (overfitting threshold), increase data enhancement strength: s e+1 =s e +Δs when ΔL val When <δ2 (undertraining threshold), reduce the data enhancement strength: s e+1 =s e -Δs where Δs is the data enhancement strength adjustment step.

[0093] Accuracy Recall F1 value TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example.

[0094] By comparing the evaluation indicators of models at different training stages on the validation set, the model with the best performance is selected. Finally, the selected model is comprehensively tested on the test set to evaluate its generalization ability and obtain the final required forklift safety control and traffic flow analysis model.

[0095] Through the above method, the technical solution provided by this embodiment has the following technical effects:

[0096] 1. Traditional data partitioning often uses simple random sampling, without considering the intrinsic distribution of data and business characteristics. This application is based on stratified sampling of forklift, cargo, and personnel data subsets to ensure that the data distribution of the training set, validation set, and test set is consistent with the actual scenario. Taking forklift data as an example, it can ensure that the sample ratio of different operating states of forklifts (such as fully loaded, unloaded, turning, etc.) in each set is reasonable, avoiding poor performance of the model in specific scenarios due to data bias, and improving the model's adaptability to actual complex working conditions.

[0097] 2. Traditional interference data processing relies solely on a single rule or a small number of feature judgments. This application integrates multi-dimensional features and complex business logic. For example, in forklift data, invalid and erroneous data is determined by analyzing the rationality of driving trajectories, the relationship between speed and operation time, and other factors. When identifying duplicate data, multiple features are comprehensively compared. This approach can more accurately identify and eliminate interference data caused by equipment failures, signal interference, data collection errors, etc., significantly improving data quality and laying a good foundation for model training.

[0098] 3. The choice of traditional model architecture is often not targeted and it is difficult to fully utilize data features. This application uses a CNN and Transformer combined architecture for the recognition of people and cargo drops at forklift stations. CNN extracts local image features, and Transformer processes long-distance dependencies, effectively capturing the detailed features and global relationships of targets in forklift operation scenarios, such as the spatial relationship between people and forks, and cargo and shelves, to improve recognition accuracy. For transport flow analysis, a network model that combines spatiotemporal features fully explores the dynamic changes in forklift spatiotemporal data through spatiotemporal convolutional layers and recurrent layers, and can more accurately reflect the changing trends of forklift traffic than traditional models.

[0099] 4. Traditional learning rate adjustment methods are single and difficult to adapt to complex training processes. This application adopts a cosine annealing learning rate adjustment strategy, dynamically adjusting the learning rate according to the number of training rounds. It converges quickly with a larger learning rate in the early stages of training, and gradually reduces the learning rate when approaching the optimal solution to avoid missing the optimal solution. Compared with traditional fixed or simple segmented learning rate adjustment methods, it can more efficiently find the global optimal solution and improve model training accuracy.

[0100] 5. Traditional data augmentation has a fixed strength, which can lead to data distortion due to over-augmentation or ineffective prevention of overfitting due to insufficient augmentation. This application dynamically adjusts the data augmentation strength based on the rate of change of validation set loss. When overfitting trends appear, the augmentation strength is increased, and when training is insufficient, the strength is reduced. This achieves adaptive adjustment of data augmentation, effectively improving the model's generalization ability and maintaining stable performance under different data distributions and scenarios.

[0101] Furthermore, in some embodiments, based on the classified model training samples, the constructed end-side deep learning model is classified and trained until the training is completed to obtain a forklift safety and control-traffic flow analysis model, including: determining sample features based on the classified model training samples, the sample features including fork-station human features, cargo drop features, and transport flow features; based on fork-station human features, cargo drop features, and transport flow features, the constructed end-side deep learning model is classified and trained until the training is completed to obtain a forklift safety and control-traffic flow analysis model.

[0102] During model training, feature extraction is inevitably required. This includes extracting features of the person using the forklift, as well as features of the cargo, including those of dropped cargo and forklift traffic. Specifically, human posture feature extraction involves extracting dynamic features of joint positions and calculating comprehensive features of joint angles.

[0103] The dynamic features of human joint positions can be calculated as follows: for the human joint positions, in the convolutional neural network (CNN) processing process, considering the dynamic changes of joint positions over time, a dynamic weight adjustment mechanism based on time series is introduced. i,t is the original position coordinate vector of the i-th joint point at time t, W i,t is the weight matrix dynamically adjusted according to time. Define the adjusted joint point position feature vector for: Where n is the number of historical time steps considered, α i,k,t is the historical time step weight coefficient that depends on time t and joint point i, which is expressed by the function g i,t Calculated, for example: It can be a function that combines the joint motion speed, acceleration, and the change in image features between the current frame and the historical frame, for example: g i,t (k)=β1·v i,t-k +β2·a i,t-k +β3·ΔF i,t-k Here i,t-k is the velocity of joint point i at time tk, a i,t-k is the acceleration, ΔF i,t-k is the feature change of the area related to the joint point in the image at time tk and time tk-1, and β1, β2, and β3 are the corresponding weight coefficients.

[0104] The comprehensive feature of joint angle can be calculated as follows: Let joint points i, j, k constitute a joint angle. The traditional method of calculating angle is relatively simple, and we innovate on this basis. Let vector and Represent the vectors from joint point i to j and from joint point j to k respectively. Define the joint point angle that comprehensively considers image context information and time dynamics for: where θ ijk is the traditional vector angle, γ is an adjustment coefficient, ΔC m,ijk is the mth context information change related to the joint angle, such as the color distribution change of surrounding pixels, texture feature change, etc. m is the weight corresponding to the context information, which is obtained through a context weight prediction model based on a deep neural network.

[0105] Furthermore, feature extraction for forklift operators also includes behavioral pattern extraction. By analyzing video sequences, the model can capture patterns in the operator's movements on the forks, such as climbing, walking, and other unusual movements. By learning these behavioral patterns, the model can more accurately distinguish between forklift operator behavior and normal work activities, improving recognition accuracy and reliability. When features matching the forklift operator's behavioral patterns are detected, an alarm is triggered and the relevant information is recorded.

[0106] Specifically, the behavior pattern characteristics can be calculated as follows: In order to accurately capture the behavior pattern characteristics of the forklift, each behavior pattern is regarded as a state sequence. Let S = {s1, s2, ..., s T} is a behavior state sequence of length T, each state s t A set of eigenvectors F t The extended form of Hidden Markov Model (HMM) is used to calculate the probability of behavior patterns.

[0107] State transition probability The current working state of the forklift (such as whether it is carrying goods, driving speed, etc.) and environmental factors (such as whether there are other forklifts around, whether the site is crowded, etc.) are taken into account. Let E be the environmental feature vector and W be the forklift working state vector, and define: where λ l is the weight coefficient, φ l (E) and ψ l (W) is a function that transforms the environmental characteristics and working status respectively.

[0108] Observation probability Not only depends on the eigenvector F t , also taking into account the time factor and the previous observation results. Let F t-1 is the eigenvector of the previous moment, defined as: where d(F t ,F t-1 ) is the eigenvector F t and F t-1 The distance metric between them, μ and δ are adjustment coefficients, b j (k) is the original observation probability.

[0109] The feature extraction of dropped goods includes the extraction of the position and shape features of the goods and the extraction of the motion trajectory features. For the position and shape features of the goods, CNN is used to analyze the cargo images captured by the video. Through training, the model can automatically learn the normal position and shape features of the goods during fork picking and transportation. In continuous video frames, the model extracts key information such as the outline, edges, and relative position of the goods with the forks and shelves. When the goods fall, these features will change significantly, such as a sudden change in the center of gravity of the goods, a reduction or disappearance of the contact points with the forks, and a distortion of the shape. By identifying these feature changes, the model determines whether the goods have fallen and records the image of the moment of falling and related time information.

[0110] Specifically, the location and shape features of the cargo can be calculated as follows: Based on CNN extraction, the spatial relationship between the cargo and the forklift, the shelf, and the influence of the surrounding environment are considered. Let P c is the center position coordinate vector of the cargo, S c is the shape description vector of the goods (e.g., shape moments calculated from contour features). The comprehensive feature vector C is defined as: where R n (P c ,S c ) represents the characteristic function of the cargo and surrounding objects such as forklifts and shelves in the nth spatial relationship, such as the relative position deviation between the cargo and the fork, the distance between the cargo and the shelf, etc. n (P c ,S c ) represents the influence function of the surrounding environmental factors on the characteristics of the goods, such as the influence of light changes on the accuracy of goods edge detection. 3,n is the corresponding weight coefficient, which is obtained by training a neural network model based on multimodal data fusion.

[0111] The weight coefficients are trained by collecting multimodal data, including cargo location, shape, spatial relationship with surrounding objects, and environmental information. This data can come from camera videos or sensor measurements from forklift operations. This data is then preprocessed, including image cropping and normalization, as well as sensor data filtering and calibration, to ensure data quality and consistency.

[0112] Next, a neural network architecture is constructed that can integrate data from multiple modalities. For example, a convolutional neural network (CNN) can be used to process image data to extract visual features such as the location and shape of the goods; a recurrent neural network (RNN) or long short-term memory network (LSTM) can be used to process sensor data to capture time series information about environmental factors and spatial relationships. To fuse the neural network branches of different modalities, the outputs of each branch can be concatenated through a connection layer and then input into a fully connected layer for further feature fusion and processing.

[0113] Then, define an appropriate loss function based on the nature of the task. For example, if it is a binary classification task for cargo drop detection, the cross entropy loss function can be used: where y i is the true label (0 means normal, 1 means the goods have fallen), is the predicted probability of the model.

[0114] Finally, an optimization algorithm (such as stochastic gradient descent SGD, Adagrad, Adadelta, Adam, etc.) is used to minimize the loss function. During the training process, the preprocessed data is input into the multimodal neural network model, the model's prediction results are calculated through forward propagation, and then the error between the prediction results and the true label is calculated according to the loss function.

[0115] Use the back propagation algorithm to propagate the error back to each layer of the network to update the weight coefficients ω1, ω2, ω 3,n Specifically, for each weight coefficient, its value is adjusted according to its corresponding gradient, so that the loss function gradually decreases. For example, in the Adam optimization algorithm, the learning rate is adaptively adjusted according to the first and second moments of the gradient to more efficiently update the weights.

[0116] To extract motion trajectory features, the model combines the target tracking algorithm with the cargo's trajectory. Under normal circumstances, the cargo's trajectory is relatively stable and conforms to the forklift's operating logic. However, when cargo is at risk of falling or has already fallen, its trajectory may exhibit abnormalities, such as sudden acceleration or deviation from the preset trajectory. By learning the characteristic patterns of normal and abnormal motion trajectories, the model can issue a timely alert at the moment cargo falls or is about to fall, and record the complete trajectory information for subsequent analysis of the cause and process of the fall.

[0117] Specifically, the motion trajectory characteristics can be calculated as follows: considering the forklift operating state (such as driving speed, whether turning, whether braking suddenly, etc.), environmental factors (such as ground flatness, light intensity, etc.), and the relationship between the cargo and the forklift (such as the way the cargo is fixed on the forklift, the center of gravity position of the cargo, etc.), the forklift operating state feature vector is set as Vt =[v t ,a t ,θ t ,…], where v t represents the speed of the forklift at time t, a t represents acceleration, θ t represents the steering angle of the forklift; the environmental factor eigenvector is E t =[e t1 ,e t2 ,…], for example, e t1 Indicates the flatness of the ground, e t2 represents the light intensity; the characteristic vector associated with the cargo and forklift is C t =[c t1 ,c t2 ,…], where c t1 Indicates the way the cargo is fixed on the forklift (such as binding, clamping, etc.), c t2 Indicates the offset of the cargo center of gravity relative to the forklift center. Introducing the state-environment-correlation adjustment function f(V t ,E t ,C t ), which is obtained through multi-layer perceptron (MLP) training and is used to comprehensively evaluate the impact of the current forklift operation status, environment, and the relationship between the cargo and the forklift on the cargo movement trajectory. The calculation formula is: Among them, β t-s It is also a weight coefficient that decreases over time, which is used to consider the impact of historical trajectories on current abnormal judgment and satisfies

[0118] if Exceeds a set dynamic threshold τ t (τ t Based on factors such as forklift operation activity and cargo type (derived through historical data statistical analysis), abnormal cargo trajectory is determined, posing a risk of dropping. Feature extraction for forklift transport flow includes features such as forklift appearance and quantity, as well as driving trajectory and speed.

[0119] Through the above solution, the technical solution provided by this embodiment has the following technical advantages:

[0120] 1. In the traditional method, the Gaussian process model is trained only based on the historical movement trajectory data of the goods. The anomaly score is calculated by the difference between the actual trajectory and the normal trajectory. It mainly focuses on the changes in the movement trajectory of the goods themselves, ignoring important information such as the forklift operating status, environmental factors, and the relationship between the goods and the forklift. In this application, the forklift operating status, environmental factors, and the relationship between the goods and the forklift are introduced, and the anomaly score is calculated by adjusting the function f(Vt ,E t ,C t ) combines these factors with the cargo's trajectory. For example, when a forklift brakes suddenly, even if the deviation in the cargo's trajectory doesn't reach the anomaly threshold under the traditional formula, the adjustment function will increase the anomaly score by factoring in this operating state, more accurately determining the risk of cargo falling. For another example, uneven ground can affect cargo's trajectory. The optimized formula can comprehensively consider these environmental factors, making anomaly detection more realistic.

[0121] 2. In the traditional method, a fixed threshold is used to determine whether the cargo movement trajectory is abnormal, which cannot adapt to different operating scenarios and working conditions. In this application, a dynamic threshold τ is used. t The dynamic threshold is adjusted based on factors such as forklift operation volume and cargo type. During busy forklift hours, the likelihood of cargo being disturbed by collisions and other disturbances increases, so the dynamic threshold is lowered accordingly, improving the sensitivity of anomaly detection. For fragile or unstable cargo, the threshold is also adjusted based on its characteristics, enhancing the pertinence and accuracy of anomaly detection.

[0122] 3. In traditional methods, the calculation logic is relatively simple, calculating the anomaly score solely by adding the Mahalanobis distance between the actual trajectory and the normal trajectory and the influence of historical trajectories. This application adds the calculation of a state-environment-correlation adjustment function. This function is trained using a multi-layer perceptron and requires learning from a large amount of forklift operation data. The calculation logic is more rigorous, but it can more accurately reflect the various factors that affect cargo movement in real-world scenarios, improving the reliability of anomaly detection.

[0123] Forklift Appearance and Quantity Characteristics: In this application, a CNN model was trained using images captured by fixed cameras at factory hubs to identify forklifts. The model automatically learned the appearance characteristics of forklifts, including body shape, color, and iconic features, to accurately distinguish forklifts from other objects. Furthermore, by detecting and counting forklifts in images, the model was able to extract quantitative characteristics of forklifts at different times. These characteristics provided fundamental data for analyzing forklift traffic within the factory.

[0124] Specifically, in this application, the appearance and quantity features of forklifts can be calculated as follows: When using CNN to extract the appearance features of forklifts, in order to more accurately distinguish forklifts from other objects, a feature enhancement method based on the attention mechanism is introduced. Let F be the original appearance feature map extracted by CNN, and an attention network is used to generate an attention weight map A. Each element a in the attention weight map A is ij Calculated by the following formula: Among them F ij is the feature vector at position (i, j) in the feature map F, f kis a set of different feature transformation functions, such as nonlinear transformation of color, texture, shape and other features, α k Is the corresponding weight coefficient. The enhanced appearance feature map F * for: In this way, the model can focus more on the key appearance features of the forklift and improve recognition accuracy.

[0125] Driving trajectory and speed characteristics: In this application, a multi-target tracking algorithm is used in combination with a deep learning model to extract features of the forklift's driving trajectory and speed. The model can track the position changes of each forklift at different time points to obtain its driving trajectory information. By calculating the change in the forklift's position in adjacent frames, the model can obtain the forklift's driving speed. These driving trajectory and speed characteristics reflect the flow of forklifts within the factory. Combined with time information, the model can analyze the changing trends of forklift traffic in different time periods, such as which time periods have large forklift traffic and which areas have slower forklift driving speeds, etc., to provide a basis for dynamically adjusting work shifts and resource allocation to avoid congestion during peak hours.

[0126] Specifically, in this application, the driving trajectory and speed characteristics can be calculated in the following way: Assume that the forklift is at different time points t1, t2, ..., t n The position coordinates are Speed ​​is Define the spatiotemporal correlation feature vector S as: in It's location and The distance between them, γ is a coefficient that adjusts the influence of speed difference, ω ij is the spatiotemporal correlation weight, obtained through training a graph neural network-based model that considers factors such as the relative positions and travel directions of forklifts. This spatiotemporal correlation feature vector S can more comprehensively reflect the flow of forklifts within the factory, providing a more robust basis for analyzing forklift traffic trends.

[0127] In this application, the acquisition method of spatiotemporal correlation weights can be obtained based on the model training of the graph neural network. The specific method can be as follows: Collect the position coordinates of the forklift at different time points speed This data can be obtained using positioning devices and sensors installed on forklifts. Each forklift is considered a node in the graph, and edges are connected based on the relative positional relationships between forklifts. For example, if two forklifts are close to each other within a certain spatial range, an edge is established between their corresponding nodes. This constructs a graph structure with forklifts as nodes and the spatial relationships between them as edges.

[0128] For each forklift node, relevant features are extracted, such as the current location, speed, travel direction, forklift type, or load, to serve as the node's initial feature vector. Edge features are calculated based on the relationships between nodes, such as the distance between two forklifts and the angle between their relative travel directions. These edge features can reflect the degree of mutual influence between forklifts.

[0129] Then, you can choose a suitable graph neural network model, such as Graph Convolutional Network (GCN), Graph Attention Network (GAT), etc. Taking GAT as an example, it can automatically learn the importance of different nodes and edges through the attention mechanism, so as to better capture the association relationship in the graph. Design a suitable model hierarchy according to the complexity of the problem and the characteristics of the data. It usually includes an input layer, several hidden layers and an output layer. The input layer is used to receive node features and edge features, the hidden layer is used to perform nonlinear transformation and feature extraction on the features, and the output layer outputs the predicted spatiotemporal association weight ω ij .

[0130] Choose a suitable loss function to measure the predicted ω ij The difference between the actual value and the true value, for example, we can define a loss function related to the actual situation according to the actual flow of forklifts, such as the frequency of encounters between forklifts and following behaviors, so that the ω learned by the model ij These phenomena can be reasonably explained. Choose appropriate optimization algorithms, such as stochastic gradient descent (SGD), Adagrad, Adadelta, etc., to minimize the loss function.

[0131] During training, gradients are calculated using the backpropagation algorithm, and the model parameters are updated using the optimization algorithm. The prepared data is divided into a training set and a validation set. The model is trained on the training set, and the model parameters are continuously adjusted to gradually reduce the loss function. The model's performance is evaluated on the validation set to prevent overfitting. If the model's performance on the validation set no longer improves, the model is considered to have converged and training is complete.

[0132] After obtaining the above features, the end-side deep learning model can be classified and trained based on these features. Different types of samples correspond to different types of models. The classification of the above features can also be based on this method. For example, for the forklift standing person feature and the drop feature, a model based on the combination of convolutional neural network (CNN) and Transformer can be trained, while the motion flow feature can be used to train a network model that combines spatiotemporal features for transport flow analysis. Of course, the end-side deep learning model can also be other models.

[0133] For example, we can also combine convolutional neural networks (CNN) with recurrent neural networks (RNN) to fully utilize the ability of CNN to extract spatial features and the advantages of RNN in processing sequence data.

[0134] Input layer: The extracted features are concatenated and used as the model input. For example, the dynamic feature vectors of joint position, the comprehensive features of joint angles, the behavioral pattern feature vectors, the comprehensive feature vectors of cargo, the forklift appearance feature map, the spatiotemporal correlation feature vectors, etc. are concatenated. Convolutional layer: Multiple convolutional layers are used to extract and abstract the input features, increasing the nonlinear expression capability of the model. Convolution kernels of different sizes, such as 3x3 and 5x5, can be used to capture features at different scales. Recurrent layer: RNN structures such as long short-term memory networks (LSTM) or gated recurrent units (GRU) are used to process time series data, such as state sequences in behavioral pattern features and cargo movement trajectory sequences. Fully connected layer: The outputs of the convolutional and recurrent layers are concatenated, and feature fusion and classification decisions are performed through multiple fully connected layers. The last layer uses the Softmax activation function to output the probability of each category.

[0135] For classification tasks such as fork standing person, cargo falling, etc., the cross entropy loss function is used, for example: Where N is the number of samples, C is the number of categories, and y i,c is the true label of the i-th sample (one-hot encoding), p i,c It is the probability predicted by the model that the i-th sample belongs to the c-th category, and the model parameters can be adjusted with optimization algorithms such as Adam to improve the recognition accuracy.

[0136] Then, the model's weights and bias parameters are randomly initialized, followed by cyclic training. Samples are selected in batches from the training samples and input into the model for forward propagation to obtain the prediction results; the loss value between the prediction results and the true labels is calculated; the backpropagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters; the optimizer is used to update the model parameters according to the gradient; this training step is repeated until the preset number of training rounds is reached, and then the obtained model is evaluated and tuned by Taurus to obtain a forklift safety control and traffic flow analysis model that meets the requirements.

[0137] Thus, by adopting the technical solution in this embodiment, the following technical benefits can be achieved:

[0138] 1. Traditional joint position extraction often only focuses on the static information of the current frame, ignoring the changes in joint position over time. However, this application can better capture the dynamic process of human movement by introducing a dynamic weight adjustment mechanism based on time series. For example, in a scene where people are standing on a fork, people will move and shake on the fork. Traditional methods are difficult to accurately track these dynamic changes. However, this application considers the number of historical time steps n and uses This item incorporates the position changes of the joint points in the past n time steps into the calculation, thereby more accurately reflecting the actual posture changes of the person on the fork.

[0139] 2. In the forklift operating environment, factors such as light and vibration will interfere with image acquisition, causing fluctuations in the detection of joint position. i,t (k) Calculate α i,k,t , which combines the joint motion velocity v i,t-k , acceleration a i,t-k And the image feature change ΔF i,t-k , which can compensate for these disturbances to a certain extent. For example, when a forklift generates vibrations during travel that affect image quality, joint position detection may deviate. However, by taking into account factors such as motion speed and acceleration, the model can determine the true trend of joint position changes based on these dynamic characteristics, rather than being misled by false changes caused by interference. This allows for more accurate identification of the joint positions of a person standing on a forklift.

[0140] 3. Traditional calculation of joint angles is based only on the vector relationship between joints, without considering image context and time dynamics. This application takes these factors into consideration, making the calculation of joint angles more comprehensive and accurate. For example, when recognizing the posture of a person standing on a fork, traditional angle calculation cannot distinguish situations with similar angles but different surrounding environments. This application uses This item incorporates contextual information such as changes in color distribution and texture features of surrounding pixels into angle calculations, which can distinguish different postures more finely.

[0141] 4. In forklift operation scenarios, different operation states (such as the type and height of the goods being transported) will change the impact of the surrounding environment on the visual presentation of human posture. For example, when transporting large goods, part of the human body will be blocked, resulting in changes in the visual relationship between joints. This application considers the change in context information ΔC m,ijk , it can still accurately calculate the joint angles in different situations, and thus more accurately identify the posture of the person standing on the fork, thereby improving the reliability of recognition.

[0142] 5. State transition probability a of traditional hidden Markov model (HMM) ij and the observation probability b j(k) The environment and time-related factors are not fully considered. In this application, the state transition probability The current working state W of the forklift and the environmental factors E are taken into account, so that the model can adapt to different working scenarios. For example, in the traditional HMM, it is impossible to distinguish the difference in the behavior patterns of people when the forklift is working in a busy warehouse and an open field. Adjusting the state transition probability can more accurately describe the transition of behavior patterns in different environments and working states. Taking into account the time factor δ·t and the previous observation results d(F t ,F t-1 ), compared with traditional methods that only rely on the current feature vector, it can better capture the temporal continuity of behavior patterns.

[0143] 6. In forklift-related scenarios, working conditions and environmental factors significantly influence human behavior patterns. For example, when a forklift is carrying heavy objects, the human on the forks will behave more cautiously and slowly, which is different from when the forklift is empty. The transition probability of the behavior pattern can be adjusted according to the working status of the forklift (whether it is carrying goods, driving speed, etc.), which is more in line with the actual situation. Taking time into account can better identify changes in personnel behavior patterns during the continuous process of forklift operations. For example, during different stages of forklift start-up, driving, and stopping, it can more accurately determine whether personnel behavior is abnormal behavior of the forklift station occupant.

[0144] 7. Traditional cargo position and shape feature extraction usually only focuses on the position and shape of the cargo itself, ignoring the spatial relationship between the cargo and surrounding objects (forklifts, shelves) and environmental impact. This application takes these factors into consideration and can provide a more comprehensive description of the cargo status. For example, when judging whether the cargo has fallen, the traditional method only relies on the change in the shape of the cargo itself, while this application uses Incorporating environmental factors such as the relative position deviation between the goods and the forklift, shelf, and lighting into the calculation of the feature vector C can more accurately determine the status of the goods.

[0145] 8. During forklift operation, the relative position and motion state of the cargo and forklift are closely related. For example, when the forklift brakes suddenly or turns, the relative position change between the cargo and the fork and the change in surrounding light will affect the judgment of cargo falling. This application considers the spatial relationship characteristic function R between cargo, forklift and shelf. n (P c ,S c ) and environmental factors influence function E n (P c ,S c), can more accurately identify whether there is a risk of cargo falling in these different forklift operation situations and issue early warnings.

[0146] 9. Traditional motion trajectory anomaly detection simply compares the deviation between the current position and the normal trajectory, without fully utilizing the historical trajectory information and considering the dynamics of trajectory changes. This application calculates the difference between the actual motion trajectory and the Gaussian process model trained based on historical data, and introduces the influence of historical trajectory on the current anomaly judgment (through ), enabling more comprehensive and accurate anomaly detection. For example, traditional methods may misjudge a trajectory that briefly deviates but then returns to normal as an anomaly. However, this application, by considering the comprehensive impact of historical trajectories, can more reasonably determine whether a trajectory is truly abnormal.

[0147] 10. In the scenario of forklift transporting goods, the acceleration, deceleration, turning and other operations of the forklift will cause the movement trajectory of the goods to show a specific change pattern. This application uses the Gaussian process model to describe the normal movement trajectory, and uses the historical trajectory weight coefficient β t-s Taking into account the historical impact of forklift operations on cargo trajectory, it is possible to more accurately detect cargo trajectory anomalies during forklift operations. For example, when the cargo shakes abnormally during a forklift turn, causing the trajectory to deviate from the normal range, this application can accurately determine whether this trajectory change is an abnormal situation of dropped cargo based on the trajectory before the turn and the forklift's operating characteristics, rather than misjudging the trajectory change caused by normal forklift operation as an anomaly.

[0148] 11. Traditional forklift appearance feature extraction is not able to distinguish forklifts from other objects in complex backgrounds. The new feature enhancement method based on attention mechanism highlights the key appearance features of forklifts through attention weight map A, which can more accurately identify forklifts in complex environments. For example, in a warehouse with a variety of equipment and goods, the appearance features extracted by traditional CNN are difficult to clearly distinguish forklifts from other objects of similar shapes. However, this application uses calculation to enhance the key appearance features of forklifts. The attention weights are obtained so that the model pays more attention to the unique features of the forklift (such as color, iconic parts, etc.), thereby improving the recognition accuracy.

[0149] 12. Forklifts have unique appearance features, such as specific body shape, color marking, etc. This application uses a function f that performs nonlinear transformation on features such as color, texture, and shape. k , which can better highlight the unique appearance characteristics of these forklifts. In forklift traffic monitoring scenarios, even in complex conditions such as changing light and partial occlusion, forklifts can be more accurately identified, providing a reliable basis for subsequent accurate forklift counting and traffic analysis.

[0150] 13. When analyzing the trajectory and speed of a forklift, traditional methods do not fully consider the temporal and spatial correlation between the trajectory and speed at different time points. This application defines a temporal and spatial correlation feature vector S to comprehensively consider the distance between the forklift position at different time points. Speed ​​Difference and the time interval (t j -t i ), can more comprehensively reflect the flow of forklifts. For example, traditional methods only analyze trajectory and speed separately, unable to discover the potential relationship between forklift speed changes and driving trajectory at different time periods, while this application can reveal these inherent connections.

[0151] 14. In the factory, the trajectory and speed of the forklift are affected by the work tasks, site layout and other forklifts. This application uses the spatiotemporal correlation weight ω obtained by graph neural network training ij This method takes into account factors such as the relative position and travel direction of forklifts, and can better adapt to the actual flow of forklifts within the factory. For example, in areas where forklifts frequently intersect, this spatiotemporal correlation feature vector S can more accurately analyze the changing trends of forklift traffic, providing stronger data support for rationally planning forklift routes and avoiding congestion, as it comprehensively considers the spatiotemporal interaction characteristics of forklifts within the factory.

[0152] S400 , obtaining a comprehensive image of a forklift to be identified, and analyzing the comprehensive image of the forklift to be identified based on a forklift safety control-traffic flow analysis model to obtain forklift safety control-traffic flow result data.

[0153] After obtaining the trained forklift safety and control-traffic flow analysis model, when obtaining the comprehensive image of the forklift to be identified again, the model can analyze the image and obtain the corresponding result data, that is, the forklift safety and control-traffic flow result data.

[0154] Specifically, at the edge, including the forklift segment, after the AI ​​vision host in the ADAS safety monitoring system receives the real-time video stream data collected by the camera, it processes the collected video stream data in a preprocessing method similar to the training stage, such as image normalization, denoising, cropping, etc., to ensure that the data quality and format input into the inference model are consistent with those during training, thereby improving the accuracy of inference.

[0155] The preprocessed data is transmitted in real time to an edge-based inference model (i.e., the forklift safety and control-traffic flow analysis model) for processing. The inference model rapidly performs calculations based on its internal parameters and structure. For forklift user recognition, the model determines whether a person is standing on the forks in the image; for cargo drop detection, the model determines whether cargo has fallen; and for transportation flow analysis, the model identifies forklifts and compiles traffic flow information to generate forklift safety and control-traffic flow analysis data.

[0156] On the data center side, for batch inference tasks (such as centralized analysis of historical video data), the data is input into the model according to the same process for inference operations, resulting in the corresponding analysis results. The inference results on the edge side are promptly fed back to the forklift's relevant monitoring system or operator terminal via a network (such as the company's internal LAN). For example, the display screen in the forklift cab displays warnings of forklift personnel, cargo drop warnings, and transportation traffic congestion information, allowing the operator to take appropriate measures in a timely manner. On the data center side, the results of batch inference can be further statistically analyzed, such as the frequency of forklift personnel incidents in different time periods, the areas and time periods with high incidence of cargo drops, and the peak and trough periods of transportation traffic in different regions. This provides data support for the company's safety management and operational process optimization. At the same time, the inference results are compared with the actual situation to provide a reference basis for subsequent model training and optimization, facilitating the continuous improvement of model performance.

[0157] Furthermore, in some embodiments, real-time model updates and deployments can be achieved through cloud-edge collaboration, and stable communication between the cloud (enterprise data center) and the edge (AI vision host on the forklift side) can be achieved by establishing a private network within the enterprise (such as a virtual private network VPN).

[0158] VPNs establish dedicated channels on public networks to ensure the security and stability of data transmission, prevent data leakage, and ensure that important data such as model updates can be accurately and promptly transmitted between the cloud and the edge. Appropriate communication protocols are selected for message delivery and data transmission between the cloud and the edge. For example, the Message Queuing Telemetry Transport Protocol (MQTT) is used to transmit messages between the edge and the cloud where bandwidth is limited and the network environment is unstable. For example, the forklift can subscribe to model update messages published by the enterprise data center and receive timely notifications when a new model version is released. The Hypertext Transfer Protocol (HTTP) is used to transmit larger data blocks, such as model files, and request methods such as POST and GET are used to download and deploy model files from the data center to the forklift.

[0159] By rationally combining these communication protocols, smooth information exchange between the cloud and the edge is ensured. When the enterprise data center collects new forklift operation data or discovers that the existing model performs poorly in certain scenarios, the data preparation and model training processes are re-performed to obtain an optimized model. Then, through the cloud-edge collaborative mechanism, the new model is deployed to the AI ​​vision host on the forklift side, achieving real-time model updates. This enables the ADAS safety monitoring system to continuously adapt to complex and changing operating environments, continuously improving the accuracy of identifying people on the forks and dropped cargo, as well as the accuracy of transportation flow analysis.

[0160] By adopting the above solution, the embodiment of the present application has the following technical advantages:

[0161] 1. Through in-depth analysis using diverse algorithms and large-scale end-to-end models, we can accurately capture forklift traffic information, including trends across time periods and regions. Compared to traditional manual statistics or simple sensing devices, this data is more accurate and comprehensive, providing a reliable basis for logistics scheduling.

[0162] 2. The logistics dispatch system dynamically adjusts work shifts and resource allocation based on forklift traffic analysis, guiding forklift routes during peak hours to avoid congestion. For example, if the system predicts impending congestion in a certain area, it will pre-plan alternative routes for forklifts, reducing wait times, improving logistics efficiency, and lowering operating costs.

[0163] 3. Through a cloud-edge collaborative mechanism, safety monitoring data and logistics scheduling data are effectively integrated. The central processing unit combines historical data with predictive models for analysis, providing strong support for management decisions. For example, by analyzing safety accident rates and forklift traffic over different time periods, work plans can be rationally adjusted to improve management efficiency.

[0164] 4. Cloud-edge collaboration enables real-time model updates, allowing the system to continuously adapt to complex and changing operating environments. As new data is collected and analyzed, the model's accuracy and adaptability continue to improve, ensuring the system remains efficient.

[0165] Furthermore, in some embodiments, forklift transport flow can be detected and identified using deep learning object detection algorithms within pre-processed video frames. These algorithms are trained on a large number of forklift images, learning feature representations of forklifts. These algorithms can accurately detect forklifts within video frames and provide their location, category, and confidence level.

[0166] For example, in one embodiment of the present application, the detection and identification of forklifts in preprocessed video frames can be performed as follows: a candidate region containing the target is generated through a region proposal network. RPN is built based on a convolutional neural network (CNN), and takes the feature map of the input image extracted by CNN as input. On the feature map, RPN uses a set of sliding windows (anchor boxes, AnchorBoxes), which have different scales (such as small, medium, and large) and aspect ratios (such as 1:1, 1:2, 2:1, etc.). For the position of each sliding window, RPN predicts the probability of the existence of a target at that position and the offset of the anchor box, which is used to adjust the position and size of the anchor box so that it surrounds the target more tightly. In this way, RPN can quickly generate a large number of candidate regions containing targets.

[0167] For the position adjustment of the anchor frame, the common position offset mode of the forklift in different working scenarios is considered. Let the center coordinate of the original anchor frame be (xa ,y a ), width is w a , height h a , the offset predicted by RPN is (Δx, Δy, Δw, Δh). After conventional calculation and adjustment, the center coordinates of the anchor box (x′ a ,y′ a ) and width and height (w′ a ,y′ a ) on the basis of (x′ a =x a +Δx·w a ,y′ a =y a +Δy·h a ,w′ a =w a ·e Δw ,h a ′=h a ·e Δh ), introducing the forklift operation scenario factor.

[0168] Assume that the feature vector of the forklift operation scene is S = [s1, s2, ..., s n ], for example, s1 indicates whether the forklift is operating in a narrow aisle (0 or 1), and s2 indicates the density of obstacles around the forklift. The offset is adjusted using a scene-dependent offset correction function f(S). The new offset is calculated as (Δx′, Δy′, Δw′, Δh′), where:

[0169]

[0170] Here α i It is the scene factor i The weight coefficient, g x ,g y ,g w ,g h is a function related to the position and size of the anchor box, such as g x (x a ,y a ,w a ,h a )=(x a -W / 2) / W (W is the image width), which indicates the influence of the relative position of the center of the anchor box in the horizontal direction of the image on the offset.

[0171] Candidate region classification and regression: After obtaining the candidate regions from RPN, these candidate regions are mapped to the feature map extracted by CNN, and the features corresponding to each candidate region are extracted. Then, two fully connected layers are used to perform target classification and bounding box regression respectively. The target classification layer predicts the category of the target in the candidate region (such as forklifts, cargo, etc.) and outputs the probability corresponding to each category; the bounding box regression layer fine-tunes the bounding box of the candidate region to more accurately locate the true position of the target. During the training process, by calculating the loss between the predicted result and the true label (such as classification loss and regression loss), the backpropagation algorithm is used to continuously adjust the network parameters so that the model can more accurately identify the target category and locate the target position.

[0172] When calculating the classification loss L cls and regression loss L reg When considering the imbalance between the forklift category and other categories, let the true category label be y true , the predicted category probability is p(y), and the category balance weight B=[b1,b2,…,b c ](c is the number of categories), where b j Represents the weight of the jth class, and the weight of the forklift class is set to b forklift , dynamically adjusted according to the ratio of the number of forklift samples to the number of samples of other categories, the cross entropy classification loss is

[0173] For regression loss, let the true bounding box coordinates be (x true ,y true ,w true ,h true ), the predicted bounding box coordinates are (x pred ,y pred ,w pred ,h pred ), considering the importance of bounding box positioning in different working states of forklift, let the forklift working state vector W = [w1,w2,…,w m ], for example, w1 represents whether the forklift is fully loaded (0 or 1), and the regression loss is adjusted by a state-related weight function q(W). The regression loss is

[0174] In one embodiment of the present application, the detection and recognition of forklifts in the preprocessed video frames can also be performed in the following manner: Image division and prediction: The input image is divided into S×S grids. For each grid, a fixed number (such as 2) of bounding boxes and their confidence levels, as well as the probability that the targets in these bounding boxes belong to different categories are directly predicted. The prediction of the bounding box includes the center coordinates, width, and height of the bounding box, and the confidence level indicates the possibility that the bounding box contains the target and the degree of match between the predicted bounding box and the real target. In this way, the target detection problem is converted into a regression problem, and the positions and categories of all targets in the image can be predicted in one forward propagation.

[0175] When calculating the bounding box confidence C, the influence of the forklift’s position in the image on the detection difficulty is considered. Let the coordinates of the center of the bounding box be (x c ,y c ), the image width is W and the height is H. The conventional confidence calculation is Where Pr(Object) is the probability that the bounding box contains the object, It is the intersection-over-union of the predicted bounding box and the true bounding box.

[0176] Introducing the position adjustment factor A(x c ,y c ), the new confidence is calculated as C′=C·A(x c ,y c ),in β1 and β2 are adjustment coefficients so that the confidence of the forklift bounding box near the center of the image can be properly enhanced, because the central area is usually detected more accurately.

[0177] The center coordinates of the predicted bounding box (x pred ,y pred ), width w pred and height h pred Consider the common size range of forklifts. Let the mean width of the forklift be μ w , with a standard deviation of σ w , the mean value of the height is μ h , with a standard deviation of σ h Based on the conventional prediction formula (assuming the prediction value is calculated by the offset and grid position output by the network), the predicted width and height are corrected:

[0178] Among them, γ1 and γ2 are adjustment coefficients used to make the predicted width and height of the forklift more consistent with the actual situation.

[0179] Feature Extraction and Prediction Network: A convolutional neural network is used to extract image features. The network typically consists of multiple convolutional and pooling layers to extract high-level semantic features from the image. At the end of the network, a fully connected layer maps the extracted features to predictions: bounding box coordinates, confidence scores, and class probabilities. During training, network parameters are optimized by calculating the loss between the predictions and the true labels. The loss function includes components such as bounding box coordinate error, confidence score error, and classification error. Because the entire image is processed at once, detection speed is very fast, making it suitable for scenarios with high real-time requirements.

[0180] When calculating the loss function, in addition to the conventional bounding box coordinate error L coord , confidence error L conf and classification error L cls , considering the importance of detecting different parts of the forklift. Assume that the position coordinate set of the key parts of the forklift in the image (such as forks, cab, etc.) is K = [(x k1 ,y k1 ),(x k2 ,y k2 ),…].

[0181] For the bounding box coordinate error, the coordinate error near the key part is given a higher weight. Let the predicted bounding box and the key part (x ki ,y ki ) is Define the weight function (α is the adjustment coefficient). The new bounding box coordinate error loss is in is the i-th bounding box coordinate error. In this way, the model will pay more attention to the accurate detection of key parts of the forklift.

[0182] Multi-target tracking algorithms (such as SORT and DeepSORT algorithms) are used to track forklifts in different frames to obtain the forklift's driving trajectory and speed information. Taking the DeepSORT algorithm as an example, the target detection algorithm is first used to detect the forklift target in the video frame, and then the appearance features of each target are extracted (such as extracting the target's feature vector through a convolutional neural network). The Kalman filter is used to predict the target's motion state. Data association is performed based on the prediction results and the detection results of the current frame. The detection results of the current frame are matched with the previously tracked targets to determine the identity of each target. By continuously updating the target's position, speed and other motion state information, continuous tracking of the forklift is achieved and the forklift's driving trajectory is obtained. At the same time, the forklift's driving speed is calculated based on the position changes of the forklift in adjacent frames.

[0183] In one embodiment of the present application, tracking of forklifts in different frames by a multi-target tracking algorithm can be performed as follows: Target detection: The multi-target detection algorithm relies on the target detection algorithm (to obtain the position and category information of the target in the video frame. In the logistics scheduling system, the target detection algorithm will identify targets such as forklifts and cargo, and output the bounding box coordinates of each target (such as the upper left and lower right corner coordinates) and the corresponding category label. These detection results serve as input to the multi-target detection algorithm.

[0184] Kalman filter prediction: Kalman filtering is the core method used by multi-target detection algorithms to predict target states. In logistics scenarios, target states include location (such as the coordinates of a forklift on a factory map), speed, acceleration, and other information. For each detected target, the Kalman filter predicts the target state of the current frame based on the target state of the previous frame. For example, based on the position and speed of the forklift in the previous frame, the possible position of the current frame is predicted. At the same time, the Kalman filter provides an uncertainty estimate of the prediction result, namely the covariance matrix, which is used to measure the reliability of the prediction.

[0185] Hungarian Algorithm Data Association: Data association is a key step in multi-target tracking, and the multi-target detection algorithm uses the Hungarian algorithm to address this problem. It determines which detection results belong to which tracked targets by calculating the similarity between the predicted target state and the target detected in the current frame. This similarity calculation is typically based on the Mahalanobis distance, which accounts for the uncertainty of the target state prediction. For example, the Mahalanobis distance between the predicted forklift position and the forklift position detected in the current frame is calculated. The smaller the distance, the more likely they are to belong to the same target. The Hungarian algorithm searches for the optimal match between all predicted and detected targets, minimizing the total matching cost, thereby completing data association and determining the corresponding relationship between each target.

[0186] State Update: Based on the data association results, the multi-target detection algorithm updates the state of the matched targets. If a predicted target successfully matches a detected target, the detected target information (such as a more accurate position) is used to correct the target state predicted by the Kalman filter, updating the target's position, velocity, and covariance matrix. For unmatched predicted targets, they are considered to have left the scene and are marked for deletion. For unmatched detected targets, a new tracker is created to begin tracking the newly appeared target.

[0187] In one embodiment, a special motion mode in a forklift operation scenario is considered. For example, when a forklift is carrying heavy objects, its speed will be reduced and its steering will be restricted. Assume that the forklift load state L k (L k =1 means full load, L k =0 means no load), a load-related state transfer matrix correction factor M(Lk ).

[0188] New state transition matrix F k =F·M(L k ),in α1 and α2 are adjustment coefficients. In this way, when the forklift is fully loaded, the position prediction will take into account the reduced speed and restricted movement, making the prediction more realistic. (Assuming a two-dimensional position ) and speed F is the traditional state transfer matrix, and the traditional prediction formula is:

[0189] Prediction covariance matrix The update also adjusts the traditional formula (Q is the process noise covariance), considering the environmental complexity E of the forklift operation area k (For example, the narrow channel environment is highly complex), let the noise adjustment matrix N(E k ). The new prediction covariance matrix is ​​updated to N(E k )=(1+βE k )I, β is the adjustment coefficient, and I is the identity matrix. In complex environments, the uncertainty of the prediction will increase, which is more in line with the actual situation.

[0190] Under the Hungarian algorithm, in logistics scenarios, features such as the color and model of the forklift also contribute to accurate association. Let the state vector of the i-th predicted forklift be The state vector of the jth detected forklift is X j , traditional Mahalanobis distance Where S is the state covariance matrix.

[0191] Introducing appearance feature distance Assume that the appearance feature vectors of the predicted forklift and the detected forklift are obtained by feature extraction method and New correlation distance ω1 and ω2 are weight coefficients. Combining these appearance features allows for more accurate differentiation between forklifts when their motion states are similar, reducing false associations.

[0192] In one embodiment of this application, a multi-target tracking algorithm can be used to track forklifts in different frames. This can also be done as follows: Object detection: The multi-target tracking algorithm uses an object detection algorithm to obtain the location and category information of objects in the video frames. Object detection identifies objects such as forklifts and cargo, and outputs bounding box coordinates and category labels.

[0193] Appearance feature extraction: Multi-target tracking algorithms incorporate appearance features to improve data association accuracy. They use deep neural networks (such as convolutional neural networks (CNNs)) to extract target appearance features. During the training phase, the CNN is trained using a large amount of image data containing a variety of targets, enabling it to learn the unique appearance characteristics of the targets. In logistics scenarios, the CNN can learn features such as the color, shape, and texture of forklifts or cargo. During actual tracking, the appearance feature vector of each detected target is extracted for subsequent data association.

[0194] The feature extraction process can be performed in the following ways:

[0195] Let the input image be I. After processing it with the convolutional neural network (CNN), the resulting feature map is F = CNN(I). Based on the traditional CNN structure, a feature enhancement module AM ​​based on the attention mechanism is introduced to address the uniqueness of the forklift's appearance.

[0196] For forklift-specific features such as color, shape, and texture, define the feature attention weight matrix W f , which is obtained by multiple iterative learning of forklift sample data. The specific calculation process is: First, calculate the similarity S between each position (i, j) in the feature map F and the typical feature vector T of the forklift (a vector composed of pre-extracted forklift color, shape and other features) ij ,For example (Here n is the feature dimension).

[0197] Then the similarity is processed by a nonlinear activation function σ (such as ReLU) to obtain A ij =σ(S ij ), thereby constructing the attention weight matrix A. The feature map F enhanced by the attention mechanism * =AM(F,A)=F⊙A (⊙ represents element-by-element multiplication).

[0198] The final extracted forklift appearance feature vector E is obtained by the enhanced feature map F * Perform global average pooling and full connection layer processing to obtain E = FC (GAP (F * )), where GAP represents the global average pooling operation and FC represents the fully connected layer.

[0199] Kalman Filter and Hungarian Algorithm: DeepSORT uses the Kalman filter for target state prediction and the Hungarian algorithm for data association, both from SORT. The Kalman filter predicts the target state, while the Hungarian algorithm calculates the similarity between the predicted and detected targets for matching. However, DeepSORT's similarity calculation not only considers the Mahalanobis distance but also incorporates the cosine distance of appearance features. This weighted fusion of the Mahalanobis and cosine distances yields a more accurate similarity metric, improving data association accuracy and reducing target ID switching.

[0200] When performing similarity calculation, assume that the target state of the forklift predicted by the Kalman filter is The covariance matrix is ​​P t , the detected forklift target state is z t , then the Mahalanobis distance d M for

[0201] Assume that the appearance feature vector of the predicted target is E p , the appearance feature vector of the detected target is E d , then the cosine distance d C for

[0202] The weight factors ω1 and ω2 related to the forklift operation scene are introduced. They are obtained by analyzing different forklift operation scenes (such as factors such as warehouse busyness and forklift density) and training using machine learning algorithms (such as decision tree regression). The final similarity measure d is: d = ω1d M +ω2d C .

[0203] Cascade matching and new target processing: The multi-target tracking algorithm uses a cascade matching strategy to handle situations such as long-term occlusion. When a target is obscured for a period of time, the uncertainty of its position prediction increases. Cascade matching first attempts to match the predicted target with the most recently detected target, and then gradually expands to matching with earlier detected targets. This prioritizes matching reliable detection results and improves tracking stability. For newly detected targets, if no match is found in the existing tracker, a new tracker is created to begin tracking. At the same time, the multi-target tracking algorithm also sets a tracker lifecycle. Trackers that have not been matched for a long time will be deleted to avoid invalid tracking occupying resources.

[0204] When performing cascade matching, it can be done in the following way. Let the prediction target set be The detection target set is {z1,z2,…,z m For each prediction target Define it and the detection target z jMatching priority P at time t ij,t First, calculate the predicted target The uncertainty factor U i,t , which is the same as the covariance matrix P in Kalman filtering i,t Related, such as U i,t =trace(P i,t )(trace represents the trace of the matrix).

[0205] For the detection target z j , considering its detection stability factor S in the latest k frames j , which can be measured by calculating the variance of the change of its bounding box coordinates in the last k frames. The smaller the variance, the higher the stability.

[0206] Matching priority P ij,t The calculation formula is Where λ1 and λ2 are adjustment coefficients. In the cascade matching process, the prediction target with the highest matching priority is preferentially selected to match with the detection target.

[0207] The large on-device model conducts in-depth analysis of collected data such as forklift quantity, location, driving trajectory, and speed, and combines it with historical traffic data and warehouse operation patterns to establish a forklift traffic prediction model. Deep learning models (such as LSTM, GRU, and Transformer) can be used to model this data and learn the time series characteristics and spatial distribution characteristics of forklift traffic. For example, the LSTM model can be used to learn from historical traffic data to capture the changing trends of traffic over time; the Transformer model can be used to learn the correlation between forklift traffic in different areas. At the same time, the model is optimized to improve prediction accuracy by considering warehouse operation patterns, such as the amount of goods in and out of the warehouse and production plans in different time periods.

[0208] To predict forklift traffic, take the LSTM model as an example. Assume that the input forklift traffic time series data is {x1,x2,…,x T}, the characteristic vector related to warehouse operation rules is R = [r1, r2, ..., r m ], for example, r1 represents the amount of goods in and out of the warehouse in the current time period, r2 represents the amount of forklift tasks arranged in the production plan, etc.

[0209] The input of the traditional LSTM model is x t , the hidden state is h t-1 , the cell state is c t-1 , through a series of gating operations (input gate i t 、Forget Gate t , output gate o t) Update the hidden state and cell state. In the model of this application, the warehouse operation regularity feature vector R is integrated into the calculation of the input gate and the forget gate.

[0210] Input gate i t =σ(W i x t +U i h t-1 +V i R+b i ), forget gate f t =σ(W f x t +U f h t-1 +V f R+b f ), where W i ,U i ,V i ,W f ,U f ,V f is the weight matrix, b i ,b f is the bias vector, σ is the sigmoid activation function. The updated hidden state h t For forklift traffic forecasting For example

[0211] Through the above-mentioned method, the technical solution provided by this embodiment has the following technical advantages:

[0212] 1. Compared to traditional appearance feature extraction, this application uses an attention mechanism to focus more on forklift-specific features such as color, shape, and texture, rather than treating all features in the image equally. In a forklift operating environment, other objects of similar shape or color are present, which can easily interfere with traditional methods. However, this application effectively highlights the key features of the forklift, improving the accuracy and robustness of appearance feature extraction, thereby more accurately identifying the forklift in subsequent data association.

[0213] 2. In similarity calculation, traditional similarity calculation cannot be flexibly adapted to different forklift operation scenarios. The weight factors ω1 and ω2 in this application can dynamically adjust the proportion of Mahalanobis distance and cosine distance in the similarity measurement according to the actual forklift operation scenario. For example, in a scenario with high density of forklifts and frequent occlusions, the weight of the cosine distance of the appearance feature can be appropriately increased, because the uncertainty of the position information increases at this time, and the appearance feature is relatively more stable; in the case of sparse forklifts and simple scenes, the weight of the Mahalanobis distance is increased, and more reliance is placed on position prediction information. This can significantly improve the accuracy of data association, reduce the situation of target ID switching, and especially perform better in different forklift operation environments.

[0214] 3. In cascade matching, traditional cascade matching is mainly based on time sequence matching. The matching priority adjustment formula of this application comprehensively considers the uncertainty of the predicted target and the stability of the detected target. In the forklift operation scenario, when the forklift reappears after being blocked, the traditional method will cause matching errors due to simple time sequence matching because the prediction uncertainty increases after being blocked. Through quantitative analysis of uncertainty and stability, this application can prioritize matching targets with low uncertainty and stable detection, greatly improving the stability and accuracy of forklift tracking under different occlusion conditions.

[0215] 4. In forklift traffic prediction, traditional forklift traffic prediction models often only focus on historical traffic data itself, ignoring the important impact of warehouse operating rules on forklift traffic. This application incorporates the warehouse operating rule feature vector into the gated calculation of the LSTM model, enabling the model to learn the intrinsic connection between operating rules and forklift traffic. For example, when the amount of goods in and out of the warehouse increases, the model can automatically adjust the prediction of forklift traffic changes based on operating rules. Compared with traditional models, it can more accurately capture the changing trend of forklift traffic, improve the accuracy of prediction, and provide a more reliable basis for warehouse logistics scheduling.

[0216] Furthermore, an embodiment of the present application also provides an industrial vehicle safety control and traffic flow analysis system based on an end-side large model, including: a sample data set acquisition module 1, used to obtain forklift comprehensive image samples, and annotate the forklift comprehensive image samples to obtain an annotated data set, the annotated data set including a forklift data subset, a cargo data subset and a forklift personnel data subset; a sample acquisition module 2, used to classify and obtain model training samples from the forklift data subset, the cargo data subset and the forklift personnel data subset based on preset model training parameters; a model training module 3, used to perform classification training on the constructed end-side deep learning model based on the classified model training samples until the training is completed to obtain a forklift safety control-traffic flow analysis model; an image analysis module 4, used to obtain a comprehensive image of the forklift to be identified, and analyze the comprehensive image of the forklift to be identified based on the forklift safety control-traffic flow analysis model to obtain forklift safety control-traffic flow result data.

[0217] Sample Dataset Acquisition Module 1 acquires comprehensive forklift image samples and annotates them to obtain an annotated dataset, which includes a forklift data subset, a cargo data subset, and a forklift operator data subset. Sample Acquisition Module 2 then classifies and acquires model training samples from the forklift data subset, cargo data subset, and forklift operator data subset based on preset model training parameters. Model Training Module 3 then performs classification training on the constructed end-side deep learning model based on the classified model training samples until the training is completed and a forklift safety and control-traffic flow analysis model is obtained. Finally, Image Parsing Module 4 acquires a comprehensive image of the forklift to be identified and analyzes it based on the forklift safety and control-traffic flow analysis model to obtain the forklift safety and control-traffic flow result data. Therefore, this application obtains model training samples from a subset of forklift personnel data, and is able to analyze the personnel on the forks. When a person is detected on the forklift forks, the security control-traffic flow analysis model can be used to timely analyze the image, facilitating timely alarms. Compared with the traditional method of only detecting whether pedestrians are close to the forklift and issuing an alarm, this application can also issue an alarm if there is a person standing on the forks, which can further improve the safety of forklift operations. This application obtains model training samples from a subset of cargo data, and is able to analyze the cargo on the forks. When cargo falls from the forks, the security control-traffic flow analysis model can be used to timely analyze the image, facilitating timely alarms. While being able to record the status of the cargo, it can also promptly alert nearby personnel, thus preventing secondary damage to the cargo as much as possible.

[0218] This application obtains model training samples from a subset of forklift data, can analyze the number and driving trajectory of forklifts, and can timely analyze images through the security control-traffic flow analysis model to promptly determine the vehicle allocation during the forklift operation process, so that relevant personnel can adjust the deployment of forklifts in a timely manner according to the allocation of forklifts and the actual conditions during driving.

Claims

1. A method for industrial vehicle safety control and traffic flow analysis based on a large end-to-end model, characterized in that: include: Obtaining forklift comprehensive image samples and annotating the forklift comprehensive image samples to obtain an annotated data set, wherein the annotated data set includes a forklift data subset, a cargo data subset, and a forklift operator data subset; Based on preset model training parameters, classify and obtain model training samples from the forklift data subset, the cargo data subset, and the forklift operator data subset; Based on the classified model training samples, the constructed end-side deep learning model is classified and trained until the training is completed to obtain a forklift safety control and traffic flow analysis model; A comprehensive image of a forklift to be identified is obtained, and the comprehensive image of the forklift to be identified is analyzed based on the forklift safety control-traffic flow analysis model to obtain forklift safety control-traffic flow result data.

2. The method according to claim 1, characterized in that The obtaining of a forklift comprehensive image sample includes: Based on the image acquisition device at the preset position, the forklift is imaged to obtain a comprehensive image sample of the forklift; The image acquisition device at the preset position includes a first image acquisition device located on the forklift and a second image acquisition device located at a hub position of the forklift route. The first image acquisition device is used to acquire images of the forks of the forklift, and the second image acquisition device is used to acquire images of the forklift.

3. The method according to claim 1, characterized in that The labeling of the forklift comprehensive image samples to obtain a labeling dataset includes: Determining personnel information, cargo information, and forklift information in the forklift comprehensive image sample; The forklift comprehensive image sample is labeled based on the personnel information, the cargo information, and the forklift information to obtain a labeled data set.

4. The method according to claim 1, wherein Before performing classification training on the device-side deep learning model, the following steps are also included: Data enhancement is performed on the model training samples, and the data enhancement includes angle transformation, size transformation and area transformation.

5. The method according to claim 1, wherein The model training samples based on classification are used to perform classification training on the constructed end-side deep learning model until the training is completed to obtain a forklift safety control-traffic flow analysis model, including: Determining sample features based on the classified model training samples, wherein the sample features include features of people standing on forks, features of cargo falling, and features of transportation flow; Based on the characteristics of the person standing on the forklift, the characteristics of the cargo falling, and the characteristics of the transportation flow, the constructed end-side deep learning model is classified and trained until the training is completed to obtain a forklift safety control-traffic flow analysis model.

6. The method according to claim 1, characterized in that The method of classifying and obtaining model training samples from the forklift data subset, the cargo data subset, and the forklift operator data subset based on preset model training parameters includes: Determining data categories of the forklift data subset, the cargo data subset, and the forklift operator data subset based on the annotation information of the annotated data set; Determining a model category of the end-side deep learning model based on preset model training parameters; The data categories are matched with the model categories, and model training samples are obtained by classification from the forklift data subset, the cargo data subset, and the forklift personnel data subset.

7. The method according to claim 6, characterized in that The obtaining of model training samples by classification from the forklift data subset, the cargo data subset, and the forklift operator data subset includes: Identify the forklift data subset, the cargo data subset, and the forklift operator data subset to obtain interference data, where the interference data includes invalid data, erroneous data, and duplicate data; The interference data is screened out and classified to obtain model training samples.

8. An industrial vehicle safety control and traffic flow analysis system based on a large end-side model, characterized in that: include: A sample data set acquisition module is used to acquire forklift comprehensive image samples and annotate the forklift comprehensive image samples to obtain an annotated data set, wherein the annotated data set includes a forklift data subset, a cargo data subset, and a forklift operator data subset; A sample acquisition module, configured to classify and acquire model training samples from the forklift data subset, the cargo data subset, and the forklift operator data subset based on preset model training parameters; A model training module is used to perform classification training on the constructed end-side deep learning model based on the classified model training samples until the training is completed to obtain a forklift safety control and traffic flow analysis model; The image analysis module is used to obtain a comprehensive image of the forklift to be identified, and analyze the comprehensive image of the forklift to be identified based on the forklift safety and control-traffic flow analysis model to obtain forklift safety and control-traffic flow result data.

Images