Forklift and pedestrian trajectory prediction and collision warning method based on multi-camera fusion

Abstract

The application provides a forklift and pedestrian trajectory prediction and collision warning method based on multi-camera fusion, and relates to the technical field of forklift and pedestrian trajectory prediction and collision warning. The application acquires panoramic and regional images in a factory building, establishes panoramic and regional detection models, detects the moving state of forklifts and pedestrians, pedestrian behavior characteristics, driver driving data and forklift characteristics in the factory building respectively, establishes a pedestrian trajectory prediction model, predicts the pedestrian walking path according to the pedestrian basic prediction data, moving state and current work content, establishes a forklift trajectory prediction model, plans the forklift carrying route according to the forklift basic prediction data, moving state and current loading and unloading content, judges the fitting degree of the forklift through real-time driving safety scoring of the driver and forklift state safety scoring, sets the warning time of the forklift, and performs collision warning according to the trajectory prediction of the pedestrian and the forklift.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of forklift and pedestrian trajectory prediction and collision warning, in particular to a forklift and pedestrian trajectory prediction and collision warning method based on multi-camera fusion. BACKGROUND

[0002] In the industrial warehouse environment, forklift and pedestrian collision accidents occur frequently. The traditional warning method relies on a single vehicle-mounted camera or sensor, which has problems such as visual blind area, high equipment cost, and complex deployment. For example, a single camera can only cover a local area in the direction of forklift operation, cannot realize full-environment monitoring, and needs to be customized and installed for different forklifts, increasing maintenance costs. In addition, existing systems are mostly based on static distance threshold warning, which is difficult to dynamically adapt to the complex motion state of forklifts and pedestrians, resulting in high risk of false positives or false negatives. Although some solutions introduce AI algorithms, they are still limited by the computing power of vehicle-mounted edge devices, making it difficult to process multi-camera fusion data in real time, and lack comprehensive modeling of pedestrian intentions and forklift states, resulting in insufficient warning accuracy. In order to improve the safety of the workplace, a system is needed that can monitor and predict the trajectories of forklifts and pedestrians in real time, accurately, and comprehensively, and provide collision warning.

[0003] In the prior art, the disclosure number CN118552933A discloses a forklift and pedestrian collision warning method and collision warning device, electronic equipment, the present application proposes a forklift and pedestrian collision warning method and collision warning device, electronic equipment and non-transitory computer readable storage medium, the collision warning method includes responding to the pedestrian collision warning instruction of the forklift, determining the warning range of the forklift; determining whether the forklift is in a motion state according to the position of the forklift in the continuous frame image; and when the forklift is in a motion state, using the warning range to warn the pedestrian collision, but this scheme only warns by identifying whether the forklift is in a motion state, and does not comprehensively consider the safety awareness of forklift drivers and pedestrians, and dynamically sets the warning time.

[0004] The above information disclosed in the background section is only used to enhance the understanding of the background of the present disclosure, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present application is to provide a forklift and pedestrian trajectory prediction and collision warning method based on multi-camera fusion to solve the problems raised in the background.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0007] A forklift and pedestrian trajectory prediction and collision warning method based on multi-camera fusion, the specific steps comprising:

[0008] Step 1: based on the plant to be prewarned, a plane coordinate is established, panoramic and regional images are obtained, a panoramic detection model is established, the moving state and identification information of the forklift and pedestrians in the plant are detected respectively, the pedestrian behavior characteristics, driver driving data and forklift characteristics are identified by establishing a regional detection model;

[0009] Step 2: the pedestrian's working information, pedestrian behavior characteristics and pedestrian passing area in the plant form the pedestrian basic prediction data, and the working content is obtained based on the semantic model, the pedestrian trajectory prediction model is established, the pedestrian historical walking trajectory is obtained, and the pedestrian walking path is predicted according to the pedestrian basic prediction data, the moving state and the current working content;

[0010] Step 3: the forklift's loading and unloading information, forklift walking characteristics, equipment parameter information and forklift driving area in the plant form the forklift basic prediction data, the loading and unloading content is obtained based on the semantic model, the forklift trajectory prediction model is established, the historical forklift carrying route is obtained, and the forklift carrying route is planned according to the forklift basic prediction data, the moving state and the current loading and unloading content;

[0011] Step 4: the historical driver driving data and safety information are obtained, the driver is scored for real-time driving safety, the forklift is scored for forklift state safety according to the forklift maintenance parameters, and the fit degree of the forklift is judged according to the driver's driving time, carrying efficiency and accident rate;

[0012] Step 5: the prewarning time of the forklift is set according to the driving safety score, the forklift state safety score, the fit degree and the forklift moving state, the safety information of the pedestrian is obtained to calculate the danger avoidance time, and the collision prewarning is performed according to the trajectory prediction of the pedestrian and the forklift, the danger avoidance time and the prewarning time.

[0013] Further, the moving state includes moving speed, moving direction and coordinate position;

[0014] The identification information includes pedestrian identity information, driver identity information and forklift number;

[0015] The pedestrian behavior characteristics include carrying state and empty-handed state;

[0016] The forklift walking characteristics include empty car state and carrying state;

[0017] The driver driving data includes turning without deceleration, sudden braking, fatigue driving, not wearing safety belt and deviating from driving area.

[0018] Further, the working information includes pedestrian identity information, working position, frequently visited position, authorized position and unauthorized position;

[0019] The work content is the current work area and the planned work area;

[0020] The step of predicting the walking path of the pedestrian is:

[0021] The pedestrian prediction data set is formed by the pedestrian basic prediction data in the trajectory process of each historical walking, the coordinate position in the moving state and the work content, and is taken as the input of the pedestrian trajectory prediction model, the moving speed, the moving direction and the walking trajectory in the moving state are taken as the output of the model, the model is trained, the pedestrian prediction data set is taken as the input of the trained model, and the moving speed, the moving direction and the walking trajectory of the pedestrian are predicted.

[0022] Further, the loading and unloading information includes driver identity information, loading and unloading site name and coordinate position;

[0023] The equipment parameter information includes forklift number, vehicle speed, load, turning radius,

[0024] The loading and unloading content includes loading site name and unloading site name;

[0025] The step of predicting the forklift path is:

[0026] The forklift prediction data set is formed by the forklift basic prediction data in the historical forklift carrying route process, the coordinate position in the moving state and the current loading and unloading content, and is taken as the input of the forklift trajectory prediction model, the moving speed, the moving direction and the forklift carrying route in the moving state are taken as the output of the model, the model is trained, the forklift prediction data set is taken as the input of the trained model, and the moving speed, the moving direction and the forklift carrying route of the forklift are predicted.

[0027] Further, the panoramic detection model and the area detection model are both based on a convolutional neural network, and the specific detection process is:

[0028] The image preprocessing method is image normalization, and the specific calculation formula is:

[0029] The image preprocessing method is to normalize the pixels of the image, and the calculation formula is:

[0030] ;

[0031] Wherein, is the normalized pixel value, is the pixel value of the panoramic and area image;

[0032] The convolutional neural network specifically includes a convolutional layer, an activation layer, a pooling layer and an output layer;

[0033] The convolutional layer extracts features from the bounding box and label of the input image through convolution operation, and the calculation formula is:

[0034] ;

[0035] wherein, is the input normalized pixel value, is the convolution kernel, is the coordinate of the output pixel matrix, is the value of the convolution kernel in the i-th row and j-th column; The calculation formula of the activation layer is:

[0036]

[0037] ;

[0038] wherein, is the coordinate of the output matrix of the convolution layer output;

[0039] The calculation formula of the pooling layer is:

[0040] ;

[0041] wherein, is the maximum pooling output;

[0042] The calculation formula of the output layer is:

[0043] ;

[0044] wherein, is the output of the pooling layer, is the result of image recognition, is the weight, is the bias parameter;

[0045] The panoramic image and the region image are respectively taken as the input of the convolution neural network, the moving state and the identification information of the forklift and the pedestrian in the factory building, the driver driving data, and the forklift feature are respectively taken as the output of the convolution neural network, and the convolution neural network is trained to obtain a panoramic detection model and a region detection model.

[0046] According to the panoramic image as the input of the panoramic detection model, the moving state and the identification information of the forklift and the pedestrian in the factory building are obtained, and according to the region image as the input of the region detection model, the pedestrian behavior feature, the driver driving data, and the forklift feature are obtained.

[0047] Further, the pedestrian trajectory prediction model and the forklift trajectory prediction model are both based on trajectory prediction, and the trajectory prediction is:

[0048] The spatial features of the local convolution target are expressed by the formula:

[0049] ​​ ;

[0050] wherein, is the convolutional feature of the pedestrian and forklift prediction dataset, is the pedestrian and forklift prediction dataset, is a convolution operation, is a convolution kernel of the prediction model;

[0051] The equation of the reset gate is expressed as:

[0052] The equation of the reset gate is expressed as:

[0053] ;

[0054] wherein, is the current on-off state of the reset gate, represents a sigmoid function, is the reset gate weight, is the reset gate bias parameter, is the hidden state of the previous time, is the convolutional feature of the current pedestrian and forklift prediction dataset;

[0055] The equation of the update gate is expressed as:

[0056] ;

[0057] wherein, is the current on-off state of the update gate, is the update gate weight, is the update gate bias parameter;

[0058] ;

[0059] wherein, is the candidate cell state, represents a hyperbolic tangent function, is the hidden state weight, is the hidden state bias parameter;

[0060] The update equation is:

[0061] ;

[0062] wherein, is the current time hidden state, is the previous time hidden state;

[0063] The attention equation is:

[0064] ;

[0065] ;

[0066] wherein, is the attention score of the hidden state at the current time, is the attention score of the hidden state at the time, is the total length of time, is the transpose of the weight vector, is the attention output;

[0067] Trajectory output equation:

[0068] ;

[0069] wherein, is the predicted target's pedestrian walking path and forklift carrying route, is the predicted target's speed is the predicted target's direction, is the output state weight, is the output state bias parameter;

[0070] wherein, the mathematical expressions of the sigmoid function and the hyperbolic tangent function are:

[0071] ;

[0072] ;

[0073] wherein, is the sigmoid function, is the sigmoid function input, is the hyperbolic tangent function, is the function input.

[0074] Further, the safety information includes safety examination score, safety warning times, and accident times;

[0075] The forklift maintenance parameters include service life, residual value ratio, and tire wear degree;

[0076] The specific steps for real-time driving safety scoring of the driver are:

[0077] The base score of each forklift driver is calculated according to the safety information, and the specific calculation formula is:

[0078] ;

[0079] wherein, is the base score of the driver, is the The safety examination score of the second driver, The number of safety examinations of the driver, The safety examination score attenuation coefficient, The deduction reduction ratio of the number of safety warnings, The number of safety warnings and the number of accidents, respectively;

[0080] According to the driving data of the driver and the basic score of the driver, the driving safety score is calculated. On the basis of the basic score of each forklift driver, once a violation type in the driving data type appears, a deduction is made according to the violation type, and the driving safety score of the driver is calculated. The score is increased for the period of time without appearing the violation driving data, and the specific calculation formula is:

[0081] ;

[0082] Among them, The driving safety score at the moment, The deduction weight of the violation type in the driving data, The deduction basic score of the violation type appearing in the driving data, The score is increased for the period of time without appearing the violation driving data, The number of violation types; According to the forklift maintenance parameters, the specific method of forklift state safety score is:

[0083] Among them, The forklift state safety score,

[0084] The service life of the forklift, The scrap life of the forklift, The residual value ratio,

[0085] The tire wear degree. Further, the specific method of judging the fitness of the forklift according to the driving time of the driver on the forklift, the handling efficiency, and the accident rate is: Among them, The fitness,

[0086] The handling efficiency, The accident rate,

[0087] The driving time of the forklift.

[0088]

[0089] ​​​​​Further, the specific method for setting the warning time of the forklift is:

[0090] ;

[0091] wherein, is the forklift deceleration coefficient, is the forklift driving speed, is the driving turning angle, is the driving safety score, is the forklift state safety score, is the cut, is the driver's basic reaction time;

[0092] The calculation method for calculating the danger avoidance duration is:

[0093] ;

[0094] wherein, is the danger avoidance duration, is the pedestrian's basic reaction time, is the safety assessment score of the second pedestrian, is the pedestrian safety assessment score decay coefficient, is the deduction reduction ratio of the number of pedestrian safety warnings, is the number of pedestrian safety warnings, respectively, and the number of accidents.

[0095] Further, the method for collision warning according to the trajectory prediction of the pedestrian and the forklift, the danger avoidance duration, and the warning time is:

[0096] ;

[0097] wherein, is the trajectory prediction of the pedestrian and the forklift, respectively;

[0098] When there is a collision position, it is determined that the forklift and the pedestrian have a collision risk;

[0099] The warning time for the pedestrian is:

[0100] ;

[0101] The warning time for the forklift is:

[0102] ;

[0103] wherein, is the warning time for the pedestrian when detecting the collision danger, is the warning time for the driver when detecting the collision danger, The forklift early warning time, The forklift early warning time,

[0104] Compared with the prior art, the beneficial effects of the present application are: the present application acquires panoramic and regional images in the factory building, establishes panoramic and regional detection models, respectively detects the moving state of the forklift and the pedestrian in the factory building, the behavior characteristics of the pedestrian, the driving data of the driver and the characteristics of the forklift, establishes a pedestrian trajectory prediction model, predicts the pedestrian walking path according to the pedestrian basic prediction data, the moving state and the current work content, establishes a forklift trajectory prediction model, plans the forklift carrying route according to the forklift basic prediction data, the moving state and the current loading and unloading content, judges the fit degree of the forklift through real-time driving safety scoring and forklift state safety scoring, sets the early warning time of the forklift, and performs collision warning according to the trajectory prediction of the pedestrian and the forklift.

[0105] According to the multi-camera fusion and deep learning technology, the present application significantly improves the trajectory prediction accuracy and the real-time of the warning. Firstly, based on the multi-camera cooperation of the panoramic detection model and the regional detection model, the dynamic capture of the forklift and the pedestrian in the factory building is realized, the problem of limited view of the traditional single camera is solved, and the risk of blind area is reduced. Secondly, combined with the multi-dimensional data of the pedestrian behavior characteristics, the forklift loading and unloading content and the driver driving behavior, the future trajectory of the pedestrian and the forklift is dynamically predicted through the hybrid model, and compared with the traditional static threshold warning method, the accuracy is improved. In addition, through real-time driving safety scoring and forklift state scoring, the system can dynamically adjust the warning threshold, for example, shorten the warning response time when the driver is tired or the forklift equipment is aging, so as to reduce the false alarm rate and improve the human-vehicle cooperation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0106] Figure 1 The present application is a whole method flowchart. DETAILED DESCRIPTION

[0107] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the present application is further described in detail below in combination with specific embodiments.

[0108] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the present application shall have the usual meaning understood by a person with ordinary skill in the art to which the present application belongs. The terms "first", "second" and similar words used in the present application do not represent any order, number or importance, but are only used to distinguish different components. The terms "include" or "contain" and similar words mean that the elements or objects appearing before the words cover the elements or objects listed after the words and their equivalents, without excluding other elements or objects. The terms "connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent relative positional relationships, which may change accordingly when the absolute position of the described object changes.

[0109] Embodiments:

[0110] Please refer to Figure 1 The present application provides a technical solution:

[0111] A forklift and pedestrian trajectory prediction and collision warning method based on multi-camera fusion, the specific steps comprising:

[0112] Step 1: Based on the factory to be warned, a plane coordinate is established, and panoramic and regional images are obtained. By establishing a panoramic detection model, the moving state and identification information of forklifts and pedestrians in the factory are detected respectively. By establishing a regional detection model, the behavior characteristics of pedestrians, driver driving data and forklift characteristics are identified.

[0113] A stable and easily identifiable reference point is selected in the factory, which will serve as the origin of the entire coordinate system. After determining the reference point, the length direction of the factory is selected as the X-axis and the width direction of the factory is selected as the Y-axis. The key positions and boundaries within the factory are measured using a laser range finder for coordinate calibration, which can ensure that the coordinates of each measurement point have a clear reference system. Using a laser range finder for measurement can improve the accuracy of positioning and reduce errors that may be caused by manual measurement, ensuring accurate calibration of key positions.

[0114] The panoramic image is a top-down planar image of the factory floor from above the factory. The image content includes the entire ground part. Through the panoramic detection model, the environment within the factory can be monitored in a large range in real time, and the positions, states and motion trajectories of forklifts and pedestrians can be captured. The advantage of this panoramic view is that it can effectively avoid the limitation of the limited field of view of traditional cameras, and can dynamically perceive the scene without relying on fixed positions, real-time acquisition of extensive information, reducing the blind area problem caused by single view or field of view limitation, thereby improving the comprehensiveness and accuracy of safety monitoring.

[0115] The regional image is mainly an object of a person and a forklift, and can identify specific behavior characteristics of the person and the forklift. The regional detection model is used to identify behavior characteristics of the person, driving data of the driver and characteristics of the forklift, which provides more refined analysis capability in the monitoring system, and further optimizes the prediction of the pedestrian path and the collision warning. The behavior analysis of the forklift can also be more accurate, combined with the moving state of the forklift, to predict the future path and operation direction of the forklift. Through the behavior characteristics of the person, the driving data of the driver and the characteristics of the forklift, the system can more intelligently judge the potential collision risk and give an early warning when danger occurs.

[0116] The traditional method often relies on a single perspective camera, which is prone to blind spots in the field of view, or can only provide limited behavior recognition, and cannot comprehensively consider the complex interaction between the forklift and the pedestrian. By combining panoramic detection and regional detection, the present scheme can realize monitoring and analysis in the whole factory, reduce missed detection and misjudgment, and improve the accuracy and timeliness of pedestrian and forklift behavior recognition. At the same time, the collision warning in the traditional method mainly depends on simple rules or physical distance calculation, while the present scheme based on deep learning and multi-data fusion can more intelligently and accurately predict the trajectory and judge the potential danger in advance, greatly improving the effectiveness and reaction time of the warning.

[0117] In the embodiment, the moving state includes moving speed, moving direction and coordinate position;

[0118] The identification information includes pedestrian identity information, driver identity information and forklift number;

[0119] The pedestrian behavior characteristics include carrying state and empty-handed state.

[0120] The forklift walking characteristics include empty car state and carrying state.

[0121] The driver driving data includes turning without deceleration, sudden braking, fatigue driving, not wearing a safety belt and deviating from the driving area.

[0122] The identification information detected by the panoramic detection model corresponds to the pedestrian identity information, the driver identity information and the forklift number identified by the regional detection model one by one, and the fusion of the data obtained from the panoramic image and the regional image is completed.

[0123] In the embodiment, the panoramic detection model and the regional detection model are both based on a convolutional neural network, and the specific detection process is as follows:

[0124] The image preprocessing method is image normalization, and the specific calculation formula is as follows:

[0125] The image preprocessing method is to normalize the pixels of the image, and the calculation formula is as follows:

[0126] ;

[0127] wherein, is the normalized pixel value, is the pixel value of the panoramic and regional image;

[0128] The convolutional neural network specifically comprises a convolutional layer, an activation layer, a pooling layer and an output layer.

[0129] The convolutional layer extracts features from the bounding box and label of the input image through a convolution operation, and the calculation formula is:

[0130] ;

[0131] wherein, is the input normalized pixel value, is the convolution kernel, is the coordinate of the output pixel matrix, is the value of the convolution kernel in the i-th row and j-th column; The calculation formula of the activation layer is:

[0132]

[0133] ;

[0134] wherein, is the coordinate of the output matrix output by the convolutional layer;

[0135] The calculation formula of the pooling layer is:

[0136] ;

[0137] wherein, is the maximum pooling output;

[0138] The calculation formula of the output layer is:

[0139] ;

[0140] wherein, is the output of the pooling layer, is the result of image recognition, is the weight, is the bias parameter;

[0141] The panoramic and regional images are respectively taken as the input of the convolutional neural network, the moving state and the pedestrian behavior feature of the forklift, the driver's driving data and the forklift feature in the factory are respectively taken as the output of the convolutional neural network, and the convolutional neural network is trained to obtain a panoramic detection model and a regional detection model.

[0142] ​​According to the panoramic image as the input of the panoramic detection model, the moving state of the forklift and the pedestrian in the factory building is obtained, and according to the regional image as the input of the regional detection model, the pedestrian behavior feature, the driver driving data and the forklift feature are obtained.

[0143] Step 2: Obtain the working information of the pedestrian, the pedestrian behavior feature and the pedestrian passing area in the factory building to form the pedestrian basic prediction data, obtain the working content based on the semantic model, establish the pedestrian trajectory prediction model, obtain the historical walking trajectory of the pedestrian, and predict the walking path of the pedestrian according to the pedestrian basic prediction data, the moving state and the current working content.

[0144] Based on the semantic model, the working content is obtained. First, the work order, the shift table and the task instruction text data in the warehouse management system are extracted, the location entity (such as the current working area, the planned area name) and the task description (such as the equipment operation range, the area switching time node) in the unstructured text are parsed through the pre-training language model (such as BERT), the spatial correlation (such as adjacent shelf area, path must pass point) and time constraint (such as planned area stay time) in the context are identified by combining the sequence labeling model, the historical working area switching logic graph (such as high-frequency path, cross-region task dependency relationship) is constructed by using the graph neural network, and finally the structured working content is output.

[0145] The working information of the pedestrian (identity information, working position, frequently visited position, authorized position and unauthorized position) provides the activity range and behavior pattern of individual pedestrians for the system. For example, after knowing the frequently visited position and authorized position of a pedestrian, the system can predict the activity area of the pedestrian in a certain time period. Through the matching of identity information, the system can accurately track the walking trajectory of each pedestrian, avoid the confusion of paths of pedestrians with different types and different responsibilities, and thus improve the accuracy of path prediction.

[0146] The working content (current working area and planned working area) provides an important clue for predicting the walking path of the pedestrian. For example, if the system knows that the pedestrian is carrying goods and the working area is close to the planned area, it can predict that the next step of the pedestrian is to move to the planned area. This information can help the system dynamically adjust the path prediction and avoid errors, especially in dynamic environments to provide real-time prediction updates.

[0147] The behavior characteristics of pedestrians (carrying state, empty-handed state, etc.) have a profound impact on path selection. In the carrying state, pedestrians usually need to choose a more stable and spacious path, and their walking speed is slower; while in the empty-handed state, pedestrians may choose a shorter and more efficient path. These characteristics help the system understand the behavior patterns of pedestrians in detail and make more accurate predictions of their movement behavior in combination with the current environment (such as the location of the goods stacking area, the width of the passage, etc.). The pedestrian traffic area in the factory provides spatial constraints for predicting the path. Through the layout of these areas and the passage of pedestrians, the system can avoid obstacle areas and calculate the most suitable path. For example, the system can avoid pedestrians in obstacle areas or dangerous areas, thereby providing the most safe and convenient path for pedestrians.

[0148] Traditional pedestrian path prediction relies on simple spatial analysis, ignoring the behavior characteristics and work content of pedestrians. However, combining individual work information, behavior characteristics, and work content for path prediction can achieve personalized path planning. This personalized prediction not only adjusts the path according to the specific task requirements of pedestrians, but also provides more accurate predictions based on their historical behavior patterns.

[0149] In this embodiment, the work information includes pedestrian identity information, work location, frequently visited location, authorized location, and unauthorized location;

[0150] The work content is the current work area and the planned work area;

[0151] The step of predicting the walking path of the pedestrian is:

[0152] The pedestrian prediction data set is formed by the pedestrian basic prediction data in the trajectory process of each walk, the coordinate position in the moving state, and the work content, which is used as the input of the pedestrian trajectory prediction model. The moving speed, moving direction, and walking trajectory in the moving state are used as the output of the model to train the model. The pedestrian prediction data set is used as the input of the trained model to predict the moving speed, moving direction, and walking trajectory of the pedestrian.

[0153] Step 3: Obtain the forklift carrying information, forklift walking characteristics, equipment parameter information, and forklift driving area in the factory to form the forklift basic prediction data. Obtain the loading and unloading content based on the semantic model, establish the forklift trajectory prediction model, obtain the historical forklift carrying route, and plan the forklift carrying route based on the forklift basic prediction data, moving state, and current loading and unloading content.

[0154] When obtaining the loading and unloading content based on the semantic model, data such as work orders, order texts, task instructions, and meeting records in the warehouse management system are obtained, and natural language processing technologies such as BERT, RoBERTa, and ALBERT are used to perform entity extraction and semantic analysis on unstructured texts such as cargo descriptions, loading and unloading addresses, and priority identifiers. Using pre-trained industry corpora, the context association and intent recognition of keywords such as cargo type, target shelf number, and operation specification are performed, and the context logic in historical loading and unloading records such as the sorting order of the same batch of goods or equipment restriction conditions is used to construct a dynamic semantic graph, and the core elements such as the urgency, cargo volume and weight restrictions in the loading and unloading task are captured, and finally the structured loading and unloading content (including target points and operation time) is output.

[0155] The forklift basic prediction data includes the driving area of the forklift, the loading and unloading information, etc., which helps to determine the starting point and ending point, driving range and possible path selection of the forklift. In the factory building, the driving area of the forklift may include explicit driving channels and obstacle areas. Through these information, the system can avoid the forklift entering the forbidden area or being affected by the goods stacking.

[0156] The moving state of the forklift includes the current driving speed, whether it is turning, whether it is in a stopped state, etc. These states directly affect the path selection of the forklift. For example, when the forklift is in a high-speed driving state, it may need to choose a more spacious road to avoid unnecessary risks caused by high-speed turning; while in a low-speed or stopped state, the forklift can more flexibly adjust the driving path.

[0157] The loading and unloading content includes the names and locations of the loading and unloading sites, which determine the specific handling tasks and targets of the forklift. According to the current loading and unloading content, the system can predict the driving path of the forklift. For example, the complexity of the loading and unloading task, the type of goods (such as heavy or light), and the layout of the loading and unloading site will all affect the path selection of the forklift. The system will automatically plan the most suitable driving route according to these information.

[0158] The equipment parameters of the forklift include speed, load capacity, and turning radius, etc., which will limit the action ability of the forklift. For example, if the forklift has too large a load capacity, its turning radius may become larger, and the system needs to avoid the forklift entering a narrow channel with a small turning radius to prevent the forklift from colliding or losing control. The speed of the vehicle will also affect the path planning, especially during the forklift driving process, too fast or too slow speed may affect the safety of driving

[0159] The real-time state, historical trajectory, loading and unloading content and equipment parameters of the forklift are comprehensively used to establish an intelligent path planning model. This makes the entire system not only be able to achieve static path planning when handling the forklift carrying task, but also be able to dynamically adjust the path and respond to the changes in the factory in real time, thereby greatly improving the intelligent level of the system.

[0160] In this embodiment, the loading and unloading information includes driver identity information, loading and unloading site name and coordinate position;

[0161] The equipment parameter information includes forklift number, vehicle speed, load, turning radius,

[0162] The loading and unloading content includes loading site name and unloading site name;

[0163] The step of predicting the forklift path is:

[0164] The forklift basic prediction data in the historical forklift carrying route process, the coordinate position in the moving state and the current loading and unloading content form a forklift prediction data set, which is used as the input of the forklift trajectory prediction model. The moving speed, moving direction and forklift carrying route in the moving state are used as the output of the model to train the model. The forklift prediction data set is used as the input of the trained model to predict the moving speed, moving direction and forklift carrying route.

[0165] The moving trajectory of pedestrians has dynamic, nonlinear and high uncertainty, and different pedestrians may have different behavior characteristics. For example, in a specific environment, pedestrians may have complex behaviors such as stopping, fast moving, turning back, etc. In order to accurately predict the walking path of pedestrians, the spatial features of the local target are convolved to extract the spatial features of pedestrians from the environment, such as the area where the pedestrian is located, the surrounding obstacles, other flows, etc., to help the system understand the current environment of the pedestrian.

[0166] Learning time series data through time sequence feature learning can learn the time dependence of pedestrians from historical walking trajectories, especially when the walking behavior of pedestrians has nonlinear patterns, which improves the prediction ability of future trajectories.

[0167] The attention equation helps the model focus on key input features, such as the influence of changes in the behavior or environment of pedestrians at certain times on path selection, thereby improving prediction accuracy.

[0168] The moving trajectory of the forklift has different characteristics from pedestrians. The driving speed, turning radius, load and other factors of the forklift determine the characteristics of its path selection. When the forklift is carrying a task, the complexity of the task and the load will affect its driving path, so special modeling is needed for the behavior characteristics of the forklift,

[0169] The spatial feature extraction of the convolution target local extracts spatial information in the forklift driving area, such as the driving channel of the forklift, obstacles, loading and unloading areas, etc., to provide spatial information of the environment for path planning.

[0170] The time sequence feature learning captures the time sequence relationship in the historical carrying route of the forklift, helping to understand the dynamic changes in the forklift path, especially during the loading and unloading process, the forklift may adjust the path due to the type of task or real-time environmental factors.

[0171] In the path planning of the forklift, the attention equation can focus on changes at key moments, such as whether the forklift is affected by obstacles, whether it needs to adjust speed, whether it needs to detour, etc., thereby improving prediction accuracy.

[0172] The motion patterns, behavior patterns and task requirements of pedestrians and forklifts are significantly different, so their trajectory prediction needs to be modeled separately. Unified processing may not fully consider their different requirements and behavior characteristics, leading to inaccurate path planning and even affecting safety.

[0173] The separate establishment of pedestrian trajectory prediction model and forklift trajectory prediction model can accurately predict and plan the paths of pedestrians and forklifts according to their different characteristics. This approach has significant advantages over existing technology, especially in improving prediction accuracy, safety, efficiency and adaptability. Through this separate modeling strategy, the optimization of the entire scheme can be effectively promoted, improving the safety and efficiency of the work environment, and ultimately achieving a more efficient and intelligent work environment.

[0174] In this embodiment, the pedestrian trajectory prediction model and the forklift trajectory prediction model are both based on trajectory prediction, and the trajectory prediction is:

[0175] The convolution target local spatial feature is expressed as:

[0176] ;

[0177] wherein, is the convolution feature of the pedestrian and forklift prediction dataset, is the pedestrian and forklift prediction dataset, is the convolution operation, is the convolution kernel of the prediction model;

[0178] The time sequence feature learning is expressed as:

[0179] The equation expression of the reset gate is:

[0180] ;

[0181] wherein, is the current on-off state of the reset gate, represents the sigmoid function, To reset the gate weights, To reset the gate bias parameters, is the hidden state at the previous moment, Convolutional features for the current pedestrian and forklift prediction dataset;

[0182] The update gate equation is:

[0183] ;

[0184] in, To update the current door switch status, To update the gate weights, To update the gate bias parameters;

[0185] ;

[0186] in, is the candidate cell state, To represent the hyperbolic tangent function, is the hidden state weight, is the hidden state bias parameter;

[0187] The update equation is:

[0188] ;

[0189] in, The hidden state at the current moment, The hidden state at the last moment;

[0190] Attention equation:

[0191] ;

[0192] ;

[0193] in, is the attention score of the hidden state at the current moment, For the The attention score of the hidden state at time t, is the total length of time, is the transpose of the weight vector, For attention output;

[0194] Trajectory output equation:

[0195] ;

[0196] in, To predict the walking path of the target pedestrian and the forklift carrying route, To predict the speed of the target To predict the direction of the target, To output the state weight, To output the state bias parameter;

[0197] Wherein, the mathematical expressions of the sigmoid function and the hyperbolic tangent function are:

[0198] ;

[0199] ;

[0200] Wherein, is a sigmoid function, is a sigmoid function input, is a hyperbolic tangent function, is a function input.

[0201] Step 4: Obtain historical driver driving data and safety information, and score the driver's real-time driving safety, score the forklift's state safety according to the forklift maintenance parameters, and judge the forklift's fit degree according to the driver's driving time, carrying efficiency and accident rate on the forklift;

[0202] In this embodiment, the safety information includes safety examination score, safety warning times, and accident times;

[0203] The forklift maintenance parameters include service life, residual value ratio, and tire wear degree;

[0204] The specific steps of scoring the driver's real-time driving safety are:

[0205] Calculate the base score of each forklift driver according to the safety information. The specific calculation formula is:

[0206] ;

[0207] Wherein, is the base score of the driver, is the safety examination score of the driver, is the safety examination times of the driver, is the safety examination score decay coefficient, is the deduction reduction ratio of safety warning times, respectively, safety warning times, and accident times;

[0208] According to the driving data of the driver and the basic score of the driver, a driving safety score is calculated. On the basis of the basic score of each forklift driver, once a violation type in the driving data type occurs, a deduction is made according to the violation type. A score is added for the absence of a violation driving data in the time period. The specific calculation formula is as follows:

[0209]

[0210]

[0211] The real-time driving safety score is calculated by collecting the driver data of the forklift driver in real time and the historical safety information of the driver (such as safety assessment score, accident record, and warning times). The score reflects the safety performance of the driver during driving. The real-time driving safety score can reflect the driving behavior of the driver in real time. The lower the score, the lower the safety awareness, and the more likely it is to cause dangerous accidents. By combining the safety state of the forklift, accurate risk warnings are provided to effectively prevent collision accidents.

[0212] The specific method for scoring the forklift state safety according to the forklift maintenance parameters is as follows:

[0213]

[0214]

[0215] The residual value ratio is the ratio of the current forklift discount price to the purchase price. The current forklift discount price is evaluated by experts. The tire wear degree is compared according to the current tire thickness and the new tire thickness.

[0216] ​​​​​​​​​​​​​​​​​​The forklift state safety score assesses the safety of the forklift itself during operation. Based on maintenance parameters of the forklift, such as service life, residual value ratio, tire wear, etc., the running state of the forklift is comprehensively evaluated. Through this score, the safety of the forklift during operation can be identified. When the state score of the forklift is low (e.g., the tires of the forklift are severely worn, the service life is long, etc.), its maneuverability and reaction speed may be affected, and there is a risk of sudden failure or delayed deceleration. In this case, in order to improve safety, the collision warning time should be appropriately extended. By giving an early warning, more time can be given to the driver to react and take safety measures, reducing the likelihood of accidents. For example, when the forklift is in a high-risk state (such as severe tire wear), the warning system will extend the time to trigger the warning, giving the driver more time to adapt to the slow response of the vehicle and ensure that pedestrians are not ignored.

[0217] In this embodiment, the specific method for judging the degree of fit of the driver to the forklift based on the length of time the driver drives the forklift, the handling efficiency, the accident rate, and the handling efficiency is:

[0218] ;

[0219] wherein, is the degree of fit, is the handling efficiency, is the accident rate, is the length of time the driver drives the forklift.

[0220] The handling efficiency is calculated by recording the time taken by each driver to drive the forklift during each handling operation, the number and weight of the goods handled, etc. From these data, the amount of goods handled per unit time can be calculated and compared with the maximum and minimum amounts of goods handled per unit time by the drivers in the factory area, thereby evaluating the handling efficiency. The accident rate is calculated based on the ratio of the total number of accidents to the total number of handling operations by the forklift drivers when driving different forklifts to handle goods. Accidents include goods falling over, goods being damaged, forklifts colliding, and scratching.

[0221] Based on the degree of fit between the driver and the forklift, the sensitivity and reaction speed of the pedestrian collision warning system can be optimized. For example, for drivers with a high accident rate or poor operation skills, a higher alert sensitivity may be required around the forklift. This can timely remind the driver to avoid potential dangers in the pedestrian area and reduce the probability of collision between pedestrians and forklifts. When the degree of fit between the driver and the forklift is low (e.g., poor operation skills or improper driving habits), the forklift may exhibit unstable driving and slow response, and therefore more frequent or earlier collision warnings are needed. Therefore, drivers with a low degree of fit may trigger an earlier alarm mechanism to avoid accidents caused by the driver's slow response or driving errors.

[0222] Step 5: set the early warning time of the forklift according to the driving safety score, the forklift state safety score, the fit degree and the forklift moving state, obtain the safety information of the pedestrian, calculate the danger avoidance time length, and perform collision warning according to the trajectory prediction of the pedestrian and the forklift, the danger avoidance time length and the early warning time.

[0223] In this embodiment, the specific method for setting the early warning time of the forklift is:

[0224] ;

[0225] wherein, is the forklift deceleration coefficient, is the forklift driving speed, is the driving turning angle, is the driving safety score, is the forklift state safety score, is the fit, is the driver's basic reaction time;

[0226] The calculation method of the danger avoidance time length is:

[0227] ;

[0228] wherein, is the danger avoidance time length, is the pedestrian's basic reaction time, is the safety score of the second pedestrian, is the pedestrian safety score attenuation coefficient, is the deduction reduction ratio of the number of pedestrian safety warnings, are respectively the number of pedestrian safety warnings and the number of accidents.

[0229] In this embodiment, the method for performing collision warning according to the trajectory prediction of the pedestrian and the forklift, the danger avoidance time length and the early warning time is:

[0230] ;

[0231] wherein, are respectively the trajectory prediction of the pedestrian and the forklift;

[0232] When there is a collision position, it is judged that the forklift and the pedestrian have a collision risk;

[0233] The early warning time for the pedestrian is:

[0234] ;

[0235] The early warning time for the forklift is: ​

[0236] ;

[0237] wherein, is the warning time for the driver when detecting the collision danger, is the warning time for the pedestrian when detecting the collision danger, is the forklift warning time, is the pedestrian danger avoidance time.

[0238] In traditional systems, the collision warning time is often fixed, which may lead to early or late warning, affecting work efficiency, and even unable to effectively avoid accidents. The present application dynamically sets the warning time by considering multiple factors such as driving safety score, forklift state safety score, and fit degree. For example, when the forklift driver's score is low, longer reaction time may be needed, and when the forklift state is poor, longer warning time needs to be set in advance. This dynamic adjustment based on state can more accurately adapt to different working environments and operating conditions, thereby improving the timeliness and accuracy of collision warning.

[0239] In the prior art, the collision warning system usually issues an alarm based on simple speed and distance judgment, ignoring the dynamic behavior of pedestrians. By taking into account the safety information of pedestrians (such as pedestrian position, motion trajectory) and danger avoidance time (i.e. pedestrian avoidance reaction time), the behavior and path of pedestrians can be more accurately predicted, and warnings can be issued in advance. This makes the system more sensitive to changes in risk in complex environments, thereby improving the reliability of the warning.

[0240] By dynamically adjusting the warning time according to the state of the forklift and the performance of the driver, the system can avoid unnecessary interference. For example, when the distance between the forklift and the pedestrian is far and the forklift is running stably, the warning time can be shorter, thereby reducing unnecessary reminders and avoiding affecting normal operation. Conversely, when the forklift state is poor or the operation is not standardized, appropriately extending the warning time can reduce the likelihood of accidents and further improve the safety and efficiency of operation.

[0241] The above formulas are dimensionless to calculate the numerical value, and the formula is obtained by software simulation of a large amount of data to obtain a formula closest to the actual situation. The preset parameters in the formula are set by a person skilled in the art according to the actual situation.

[0242] The above-described embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the above-described embodiments can be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art can be aware that units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solutions.

[0243] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, and can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiments according to actual needs.

[0244] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A forklift and pedestrian trajectory prediction and collision warning method based on multi-camera fusion, characterized in that, The specific steps include: Step 1: based on the plant to be prewarned, a plane coordinate is established, panoramic and regional images are obtained, a panoramic detection model is established, the moving state and identification information of the forklift and pedestrians in the plant are detected respectively, the pedestrian behavior characteristics, driver driving data and forklift characteristics are identified by establishing a regional detection model; Step 2: the pedestrian's working information, pedestrian behavior characteristics and pedestrian passing area in the plant form the pedestrian basic prediction data, and the working content is obtained based on the semantic model, the pedestrian trajectory prediction model is established, the historical walking trajectory of the pedestrian is obtained, and the pedestrian walking path is predicted according to the pedestrian basic prediction data, the moving state and the current working content; Step 3: the forklift's loading and unloading information, forklift walking characteristics, equipment parameter information and forklift driving area in the plant form the forklift basic prediction data, the loading and unloading content is obtained based on the semantic model, the forklift trajectory prediction model is established, the historical forklift carrying route is obtained, and the forklift carrying route is planned according to the forklift basic prediction data, the moving state and the current loading and unloading content; Step 4: the historical driver driving data and safety information are obtained, the driver is scored in real time for driving safety, the forklift is scored for forklift state safety according to the forklift maintenance parameters, and the fit degree of the forklift is judged according to the driving time, carrying efficiency and accident rate of the driver to the forklift; Step 5: the warning time of the forklift is set according to the driving safety score, the forklift state safety score, the fit degree and the moving state of the forklift, the safety information of the pedestrian is obtained to calculate the danger avoidance time, and the collision warning is carried out according to the trajectory prediction of the pedestrian and the forklift, the danger avoidance time and the warning time; The safety information includes safety examination score, safety examination times, safety warning times and accident times; The forklift maintenance parameters include service life, residual value ratio and tire wear degree; The specific steps for scoring the driver in real time for driving safety are: The basic score of each forklift driver is calculated according to the safety information, and the specific calculation formula is: wherein, is the basic score of the driver, is is the safety examination score of the second driver, is the number of safety examinations of the driver, is the safety examination score decay coefficient, is the deduction reduction ratio of the number of safety warnings, are the number of safety warnings and the number of accidents, respectively; According to the driving data of the driver and the basic score of the driver, a driving safety score is calculated. On the basis of the basic score of each forklift driver, once a violation type in the driving data type occurs, a deduction is made according to the violation type. The driving safety score is increased for the period in which no violation driving data occurs. The specific calculation formula is as follows: time period. The specific calculation formula is as follows: wherein, is a driving safety score at a moment, is a deduction weight of a rule type in the driving data, is a deduction base score of the rule type appearing in the driving data, is a deduction base score of the rule type appearing in the driving data, is a score for the driving data not appearing the rule type in the time period, is a score for the driving data not appearing the rule type in the time period, is a number of the rule types; The specific method for scoring the forklift for forklift state safety according to the forklift maintenance parameters is: wherein, is a forklift status safety score, is a forklift age, is a forklift retirement age, is a residual value ratio, is a tire wear degree; The specific method for judging the fit degree of the forklift according to the driving time, carrying efficiency and accident rate of the driver to the forklift is: wherein, is the fit, is the handling efficiency, is the accident rate, is the driving time on the fork truck.

2. The method according to claim 1, wherein the method is characterized in that: The moving state includes moving speed, moving direction and coordinate position; The identification information includes pedestrian identity information, driver identity information and forklift number; The pedestrian behavior characteristics include carrying state and empty-handed state; The forklift walking characteristics include empty car state and carrying state; The driver driving data includes turning without deceleration, sudden braking, fatigue driving, not wearing safety belt and deviating from the driving area.

3. The method of claim 1, wherein the method is based on multi-camera fusion. The working information includes pedestrian identity information, working position, frequently visited position, authorized position and unauthorized position; The working content is the current working area and the planned working area; The steps for predicting the pedestrian walking path are: The pedestrian prediction data set is formed by pedestrian basic prediction data in each historical walking track process, coordinate positions in a moving state and work content, taken as input of a pedestrian track prediction model, and the moving speed, moving direction and walking track in the moving state are taken as output of the model to train the model, and the pedestrian prediction data set is taken as input of the trained model to predict the moving speed, moving direction and walking track of the pedestrian.

4. The method of claim 1, wherein the method is based on multi-camera fusion. The loading and unloading information includes driver identity information, loading and unloading site name and coordinate position; The equipment parameter information includes forklift number, vehicle speed, load capacity and turning radius, The loading and unloading content includes loading site name and unloading site name; The steps of predicting the forklift path are: The forklift prediction data set is formed by forklift basic prediction data in each historical forklift carrying route process, coordinate positions in a moving state and current loading and unloading content, taken as input of a forklift track prediction model, and the moving speed, moving direction and forklift carrying route in the moving state are taken as output of the model to train the model, and the forklift prediction data set is taken as input of the trained model to predict the moving speed, moving direction and forklift carrying route of the forklift.

5. The method of claim 1, wherein the method is based on multi-camera fusion. The panoramic detection model and the area detection model are both based on a convolutional neural network, and the specific detection process is: The image preprocessing method is image normalization, and the specific calculation formula is: The image preprocessing method is to normalize the pixels of the image, and the calculation formula is: wherein, is the normalized pixel value, is the pixel value of the panorama and the region image; The convolutional neural network specifically comprises a convolutional layer, an activation layer, a pooling layer and an output layer; The convolutional layer extracts features from the bounding box and label of the input image through convolution operation, and the calculation formula is: wherein, is an input normalized pixel value, is a convolution kernel, is a coordinate of an output pixel matrix, is a value of the convolution kernel in the row and the column. The calculation formula of the activation layer is: wherein, is the coordinate of the output matrix of the convolution layer; The calculation formula of the pooling layer is: wherein, is the max-pooling output; The calculation formula of the output layer is: wherein, is the output of the pooling layer, is the result of the image recognition, is the weight, is the bias parameter; The panoramic and area images are taken as input of the convolutional neural network respectively, and the moving state, identification information and pedestrian behavior characteristics of the forklift and pedestrian in the factory, driver driving data and forklift characteristics are taken as output of the convolutional neural network respectively to train the convolutional neural network, and the panoramic detection model and the area detection model are obtained respectively; The moving state, identification information of the forklift and pedestrian in the factory are obtained by taking the panoramic image as input of the panoramic detection model, and the pedestrian behavior characteristics, driver driving data and forklift characteristics are obtained by taking the area image as input of the area detection model.

6. The method of claim 1, wherein the method is based on multi-camera fusion. The pedestrian track prediction model and the forklift track prediction model are both based on track prediction, and the track prediction is: The spatial features of the local convolutional target are expressed by a formula: wherein, a convolutional feature for a prediction dataset of pedestrians and forklifts, a prediction dataset for pedestrians and forklifts, a convolutional operation, a convolutional kernel for a prediction model; The time sequence feature learning is expressed by a formula: The equation expression of the reset gate is: wherein, to reset the current on-off state of the gate, denotes a sigmoid function, is a reset gate weight, is a reset gate bias parameter, is the hidden state at the previous time step, is the convolutional feature of the current pedestrian and forklift prediction dataset; The equation expression of the update gate is: wherein, to update the current on-off state of the gate, to update the gate weight, to update the gate bias parameter; wherein, is a candidate cell state, is a hyperbolic tangent function, is a hidden state weight, is a hidden state bias parameter; The update equation is: wherein, is the hidden state at the current time instant, is the hidden state at the previous time instant; The attention equation is: wherein, is the attention score for the hidden state at the current time instant, is the attention score for the hidden state at the time instant, is the total length of time, is the transpose of the weight vector, is the attention output; The track output equation is: wherein, is a predicted target's walking path and forklift carrying route, is a predicted target's speed is a predicted target's direction, is an output state weight, is an output state bias parameter; The mathematical expressions of the sigmoid function and the hyperbolic tangent function are: wherein, is a sigmoid function, is a sigmoid function input, is a hyperbolic tangent function, is a function input.

7. The method of claim 1, wherein the method is based on multi-camera fusion. The specific method of setting the forklift warning time is: wherein, is a forklift deceleration coefficient, is a forklift travel speed, is a travel turning angle, is a travel safety score, is a forklift state safety score, is a cut, is a driver basic reaction time; The calculation method of the danger avoidance time length is: wherein, is a danger avoidance duration, is a pedestrian base reaction time, is is a safety assessment score of the second pedestrian, is a safety assessment score decay coefficient of the pedestrian, is a deduction reduction ratio of the safety warning times of the pedestrian, are respectively the safety warning times and the accident times of the pedestrian.

8. The method of claim 1, wherein the method is based on multi-camera fusion. The collision warning method based on the track prediction of the pedestrian and the forklift, the danger avoidance time length and the warning time is: wherein, are trajectory predictions for a pedestrian and a forklift, respectively; When there is a collision position, it is judged that the forklift and the pedestrian have a collision risk; The warning time for the pedestrian is: The warning time for the forklift is: wherein, is the warning time for the driver when a collision danger is detected, is the warning time for the driver when a collision danger is detected, is the forklift warning time, is the pedestrian danger avoidance time.

Images