Forklift driver simulation training method and system

Abstract

The invention relates to the technical field of intelligent teaching, in particular to a forklift driver simulation training method and system, and provides a scene simulation mode and a free simulation mode, so that the requirements of structured training are met, and autonomous practice is supported. In the scene simulation mode, data such as user operation types and operation quantities are collected in real time, the data are matched and compared with standard data based on a time sequence, and real-time operation suggestions are generated, so that the problem that students cannot adjust operations in real time due to the fact that evaluation is performed only after existing scene simulation is finished is solved; in the free simulation mode, full-process operation data are collected to form a time sequence data set, then loading, unloading and other scene operations are recognized through a dynamic window and a pre-training scene recognition model, and finally corresponding standard data are matched to generate suggestions, so that the blank of lack of a free practice mode in the prior art is filled, and the training efficiency is improved. And the problem of no targeted evaluation after free operation is solved, flexible adaptation of a training mode is finally realized, and the operation standardization and the actual operation joining ability of students are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent teaching, in particular to a forklift driver simulation training method and system. BACKGROUND

[0002] With the upgrading of the logistics and warehousing industry, the forklift as the core handling equipment, the standardized operation of its driver directly affects the work efficiency and safety, but the traditional forklift training has the pain points of high practical operation risk and high cost, and the simulation training device gradually becomes the mainstream. However, the existing forklift driver simulation training technology still has significant defects: (1) The training mode is single and fragmented, most simulation systems only support operation simulation of fixed business scenarios (such as single loading or unloading scenarios), lack of free practice mode, which makes students unable to independently familiarize with the operation logic in an environment without pre-set constraints, and difficult to adapt to the variable scenarios in actual work; Some systems that support free mode do not establish an association mechanism between operation data and business scenarios, so students cannot obtain targeted evaluation after free operation, and the effect of independent practice is greatly reduced.

[0003] (2) The existing scene simulation system mostly generates an overall evaluation report only after the operation is completed, and does not collect user operation data in real time based on time sequence and provide real-time guidance and suggestions.

[0004] Therefore, there is an urgent need for a new intelligent training scheme to provide more free and intelligent training guidance. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a forklift driver simulation training method and system that can provide more free and intelligent training guidance.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is: A forklift driver simulation training method, comprising the steps of: According to the user's request, control the simulation device to enter the scene simulation mode or the free simulation mode; If it is a scene simulation mode, execute the following steps: A1, real-time collection of each user operation data, the user operation data including operation type and user operation amount; A2, matching and comparing the user operation data with the preset standard data based on time sequence, and generating real-time operation suggestions according to the comparison result; If it is a free simulation mode, execute the following steps: B1, real-time collection of user operation data during the free simulation mode, and generation of operation time sequence data set formed by the user operation data after the free simulation mode ends; B2, based on the operation timing data set, using a dynamic window and a pre-trained scene recognition model, identify the scene operation therein; B3, for each of the identified scene operation, match and compare the corresponding scene simulation mode standard data, and generate operation suggestions according to the comparison result.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: A forklift driver simulation training system, comprising a data acquisition module, a timing matching module, a scene recognition module and a prompt generation module, and the steps of the above-mentioned forklift driver simulation training method are realized by cooperation of the data acquisition module, the timing matching module, the scene recognition module and the prompt generation module.

[0008] The beneficial effects of the present application are that the forklift driver simulation training method and system provides two modes of scene simulation and free simulation, which meets the needs of structured training and supports autonomous practice; in the scene simulation mode, real-time collection of user operation type and operation amount data, based on timing, matching and comparing with standard data and generating real-time operation suggestions, solves the problem that the existing scene simulation only evaluates after the end and the student cannot adjust the operation immediately; in the free simulation mode, first collect full-process operation data to form a timing data set, then identify loading, unloading and other scene operations through a dynamic window and a pre-trained scene recognition model, and finally match the corresponding standard data to generate suggestions, which not only fills the gap of the lack of free practice mode in the prior art, but also solves the problem of non-targeted evaluation after free operation, and finally realizes flexible adaptation of the training mode and improves the student's operation standardization and actual operation connection ability. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 A flowchart of a forklift driver simulation training method according to an embodiment of the present application; Figure 2 A schematic diagram of a forklift driver simulation training system according to an embodiment of the present application; REFERENCE NUMERALS 1, a forklift driver simulation training system; 2, a data acquisition module; 3, a timing matching module; 4, a scene recognition module; 5, a prompt generation module. DETAILED DESCRIPTION

[0010] In order to explain the technical content, the purpose and effect of the present application in detail, the following will be explained in conjunction with the embodiments and the drawings.

[0011] The forklift driver simulation training method and system described above are suitable for assisting simulation training of forklift driver students, which will be described in detail through specific embodiments. Referring to Figure 1 An embodiment of the present application is: A forklift driver simulation training method, comprising the steps of: According to the user's request, control the simulation equipment to enter the scene simulation mode or the free simulation mode; If it is a scene simulation mode, then execute the following steps: A1, real-time collection of each user operation data, the user operation data includes operation type and user operation amount.

[0012] In this embodiment, the collection of operation data is based on a fixed collection frequency of 10-20Hz, and the minimum action component collected contains two types of core data: Operation type: specifically the basic action type of the forklift, such as mast lifting, fork tilting, steering wheel turning, driving (forward / backward), throttle / brake, and fork extension, which is determined by the action trigger signal of the sensor.

[0013] User operation amount: quantitative parameters corresponding to each type of operation, such as displacement value (unit: mm) of mast lifting, angle value (unit: °) of fork tilting, angular velocity (unit: ° / s) of steering wheel turning, driving speed (unit: km / h), throttle pedal stroke (unit: %), etc., which is read in real time by a special sensor.

[0014] The collected raw data will be filtered for noise (sensor jitter error is eliminated by Kalman filtering algorithm), and time stamp synchronization (a unified time axis label is bound to each action component, with an error of ≤10ms).

[0015] A2, match and compare the user operation data with the preset standard data based on time sequence, and generate real-time operation suggestions according to the comparison results.

[0016] In this embodiment, the standard data is composed of a plurality of sequentially arranged standard operation steps, each of which includes operation type and standard operation amount.

[0017] Step A2 includes the following steps: A21, integrate the user operation data that is continuous and has the same operation type, to generate a user operation step that includes the operation type and cumulative operation amount.

[0018] In this embodiment, when the operation type is consistent within the continuous collection period (≥3 periods), and the change trend of the operation amount has no reverse fluctuation (such as continuous mast lifting without downward movement), the system automatically starts the integration process.

[0019] Taking the gantry lifting operation as an example, if the lifting displacements of 5 consecutive collection cycles (collection frequency 10 Hz, i.e. 0.5 s) are 20 mm, 22 mm, 18 mm, 21 mm and 19 mm respectively, the integrated user operation step is “gantry lifting”+“cumulative operation amount 100 mm”, and the duration of this step (0.5 s) is recorded.

[0020] A22, based on the timing matching of the standard operation step corresponding to the user operation step, the cumulative operation amount and the standard operation amount are compared to generate an operation amount prompt for the user, and when the cumulative operation amount meets the standard operation amount, the next standard operation step is generated for the user according to the operation type prompt.

[0021] In this embodiment, dynamic time warping (DTW) algorithm is used to realize the timing alignment of user operation step and standard operation step, which can be compatible with the difference between fast and slow user operations (for example, a user completes a step in 1 s, and the standard step takes 0.8 s), and the matching threshold is set to 0.15 (i.e. timing deviation ≤15% is determined as effective matching).

[0022] When the deviation of cumulative operation amount and standard operation amount exceeds ±10%, real-time prompt is triggered: for example, the standard gantry lifting height is 100 mm, and the user cumulative operation amount reaches 115 mm. The system will prompt through the joystick vibration + voice broadcast “the gantry lowering height has exceeded the standard, please reduce the lowering amplitude by 5-10 mm”; if the deviation is ≤10%, it is determined that the operation amount meets the standard.

[0023] When the operation amount of the current step meets the standard, the system will prompt the next standard operation step through the screen pop-up window + voice 1 collection cycle in advance, for example, after the current gantry lifting step meets the standard, the pop-up window displays “the next step is to perform the fork front tilting operation”, and the voice is broadcasted simultaneously, guiding the user to operate according to the standard timing.

[0024] If it is a free simulation mode, the following steps are performed: B1, real-time collection of user operation data during the free simulation mode, and generation of operation timing data set formed by the user operation data after the free simulation mode ends; B2, based on the operation timing data set, using dynamic window and pre-trained scene recognition model to identify the scene operation therein; Step B2 includes the following steps: B21, integrating the user operation data which is continuous and has the same operation type into a user operation step including the operation type and cumulative operation amount, to obtain a user step sequence.

[0025] In this embodiment, the user operation data of the whole process is integrated according to the integration rule of step A21 to obtain user step data, and the sequence data is composed of user steps.

[0026] B22, constructing a dynamic window, dynamically adjusting the window length of the dynamic window according to the number of user operation steps involved in a preset time length.

[0027] In this embodiment, the preset time length is 6s, the minimum window length is 2 operation steps, and the maximum window length is 6 operation steps, so as to avoid confusion of scene features due to too long window or missing features due to too short window.

[0028] If the number of user operation steps collected within 6s is 2 (such as only “driving + gantry lifting” is completed), the window length of the dynamic window is set to 2; If 4 operation steps are collected within 6s (such as “driving + gantry lifting + fork tilting + fork extension”), the window length of the dynamic window is adaptively adjusted to 4; If the number of steps exceeds 6, the dynamic window is split to ensure that the length (number of steps) of each dynamic window does not exceed the maximum value.

[0029] The window slides by 1 operation step to ensure that there is no missing operation segment, for example, the previous window covers steps 1-4, and the next window covers steps 2-5.

[0030] B23, after the user operation steps covered by each dynamic window are extracted, the pre-trained scene recognition model is input, and the scene operation in the scene recognition model is identified.

[0031] In this embodiment, the user operation steps in each dynamic window are combined with the user operation data to extract features: (1) operation type combination feature (dimension: 7xN, N is the number of steps in the window): There are 7 types of forklift operation types (gantry lifting, fork tilting / tilting, steering, driving, throttle / brake, fork extension), and the extraction logic is: The operation type of each step is one-hot encoded: for example, “gantry lifting” is encoded as [1, 0, 0, 0, 0, 0, 0], and “fork tilting” is encoded as [0, 1, 0, 0, 0, 0, 0]; The one-hot encoding vectors of N steps are spliced in time sequence to form an operation type combination feature matrix (for example, a window contains 3 steps, and the dimension is 7x3).

[0032] (2) time sequence correlation feature (dimension: N-1): Reflect the time logic between steps, and the extraction logic is: Calculate the time interval of the adjacent steps: interval t = start time of step i+1 - end time of step i; Normalize (map to [0, 1]) all time intervals and splice into a one-dimensional vector (for example, 2 interval values are generated for 3 steps, and the dimension is 2); If the steps are continuous and non-interval operations (such as continuous driving + gantry lifting), the interval value is 0.

[0033] (3) Operation amount interval feature (dimension: N): Fusion of original “stationarity feature”, extraction logic: Take the normalized operation amount value of each step to form a basic vector; Superimpose the stationarity feature: calculate the volatility of the operation amount in this step (volatility = operation amount standard deviation / operation amount mean), if the volatility > 0.1 (i.e. operation amount jitter is obvious), then multiply the operation amount value at the corresponding position by 0.8 (weaken the feature weight of unstable operation through weight); Finally form a one-dimensional vector (for example, the dimension is 3 for 3 steps).

[0034] Splice the above three types of features in the order of “operation type combination feature (flattened to one dimension) + time sequence correlation feature + operation amount interval feature” to form a unified input vector.

[0035] The scene recognition model adopts a CNN-LSTM model; The CNN-LSTM model includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, an LSTM layer, a first fully connected layer, a Dropout layer, a second fully connected layer, and an output layer. The activation function used by the first convolutional layer, the second convolutional layer, and the first fully connected layer is the ReLU activation function, and the activation function used by the second fully connected layer is the Softmax activation function.

[0036] The input dimension of the input layer is dynamically adapted (26-44); First convolutional layer (CNN): convolution kernel: 32, size 3x3, step 1, activation function: ReLU; First pooling layer: maximum pooling, pooling kernel 2x2, step 1; Second convolutional layer (CNN): convolution kernel: 64, size 2x2, step 1, activation function: ReLU; Second pooling layer: maximum pooling, pooling kernel 2x2, step 1; LSTM layer: hidden layer dimension: 128, single layer, return sequence = False, dropout = 0.2; First fully connected layer: number of neurons: 64, activation function: ReLU; Dropout layer: dropout = 0.2; Second fully connected layer: number of neurons: number of scene operation categories (e.g. 12 categories: loading, unloading, stacking, narrow lane driving, etc.), activation function: Softmax; Output layer.

[0037] The training data of the scene recognition model is obtained by collecting the operation steps of professional operators, novice operators and students performing corresponding scene simulation modes.

[0038] For example, 1000 groups of operation sequences of forklift typical scenes of different proficiency personnel (300 groups of loading, unloading and stacking each, and 100 groups of other scenes) are labeled, and the training set / validation set / test set is divided according to 7:2:1.

[0039] Optimizer: Adam, learning rate = 0.001, weight decay = 1e-5.

[0040] Loss function: the loss function adopted by the CNN-LSTM model is a weighted cross-entropy loss function, which is represented as: ; ; wherein, C represents the number of categories of scene operations, y i represents the true label of the i-th scene operation, p i represents the confidence of the i-th scene operation output by the model Softmax, w i represents the weighted weight of the i-th scene operation, N total represents the total number of training set samples, N i represents the number of samples of the i-th scene operation.

[0041] Training strategy: batch size (batch_size) = 32, number of iterations = 100 rounds, early stopping strategy (stop training if the validation set loss increases for 5 consecutive rounds).

[0042] Model deployment: after training, quantize to ONNX format and deploy to forklift simulation training terminal (computing power requirement: CPU ≥ 4 cores, memory ≥ 8G, no GPU required).

[0043] B3, for each of the identified scene operations, the standard data of the corresponding scene simulation mode is obtained, matched and compared, and operation suggestions are generated according to the comparison results.

[0044] In this embodiment, for the identified scene operation, the standard operation step sequence corresponding to the scene is called, and the DTW algorithm is used for full sequence matching to count the operation amount deviation and timing deviation of each step.

[0045] In addition to the opinion generation as in step A22, a visual evaluation report can also be generated in this embodiment, including: The compliance rate of scene operation (such as the loading scene operation compliance rate 82%); Specific deviation points (such as "the fork front inclination angle reaches 8°, which exceeds the standard value by 3°, and is easy to cause the goods to slide off"); Targeted optimization suggestions (such as "it is suggested that when the fork front inclination operation is performed, the angle should be controlled within the range of 3°-5°, and the scale mark of the joystick can be used for auxiliary judgment").

[0046] According to another aspect of the present application, Figure 2 is a schematic diagram showing a forklift driver simulation training system according to an embodiment of the present application. Embodiment two of the present application is: A forklift driver simulation training system 1, comprising a data acquisition module 2, a timing matching module 3, a scene recognition module 4 and a prompt generation module 5, and realizing the steps in the forklift driver simulation training method described in embodiment one by cooperation of the data acquisition module 2, the timing matching module 3, the scene recognition module 4 and the prompt generation module 5.

[0047] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent transformation or direct or indirect application in related technical fields based on the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A method of forklift driver simulation training, characterized by, The method comprises the steps of: controlling the simulation device to enter a scene simulation mode or a free simulation mode according to a user request; if the scene simulation mode, performing the steps of: A1, collecting real-time user operation data, the user operation data comprising an operation type and a user operation amount; A2, matching and comparing the user operation data with preset standard data based on time sequence, and generating real-time operation suggestions according to the comparison result; if the free simulation mode, performing the steps of: B1, collecting real-time user operation data during the free simulation mode, and generating operation time sequence data sets formed by the user operation data after the free simulation mode ends; B2, identifying scene operations in the operation time sequence data sets based on the operation time sequence data sets, using a dynamic window and a pre-trained scene recognition model; B3, for each of the identified scene operations, matching and comparing corresponding standard data of the scene simulation mode, and generating operation suggestions according to the comparison result.

2. The method of claim 1, wherein, The standard data is composed of a plurality of sequentially arranged standard operation steps, each of which comprises an operation type and a standard operation amount.

3. The method of claim 2, wherein, Step A2 comprises the steps of: A21, integrating the user operation data that is continuous and has the same operation type to generate a user operation step comprising the operation type and a cumulative operation amount; A22, matching the standard operation step corresponding to the user operation step based on time sequence, generating an operation amount prompt for the user according to the comparison between the cumulative operation amount and the standard operation amount, and generating an operation type prompt for the user according to the next standard operation step when the cumulative operation amount meets the standard operation amount.

4. The method of claim 2, wherein, Step B2 comprises the steps of: B21, integrating the user operation data that is continuous and has the same operation type into a user operation step comprising the operation type and a cumulative operation amount to obtain a user step sequence; B22, constructing a dynamic window, and dynamically adjusting the window length of the dynamic window according to the number of user operation steps involved within a preset time length; B23, inputting the user operation step covered by each dynamic window into a pre-trained scene recognition model after feature extraction, and identifying scene operations in the user operation step by the scene recognition model.

5. The method of claim 4, wherein, The scene recognition model adopts a CNN-LSTM model. The CNN-LSTM model comprises an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, an LSTM layer, a first full connection layer, a Dropout layer, a second full connection layer, and an output layer.

6. The method of claim 5, wherein, The loss function adopted by the CNN-LSTM model is a weighted cross-entropy loss function.

7. The method of claim 6, wherein, The weighted cross-entropy loss function is expressed as: ； ； wherein, C represents the number of categories of scene operations, y i represents the true label of the i-th category of scene operation, p i represents the confidence of the i-th category of scene operation in the model Softmax output, w i represents the weighted weight of the i-th category of scene operation, N total represents the total number of training set samples, N i represents the number of samples of the i-th category of scene operation.

8. The method of claim 5, wherein, The activation functions adopted by the first convolutional layer, the second convolutional layer, and the first full connection layer are ReLU activation functions, and the activation function adopted by the second full connection layer is a Softmax activation function.

9. The method of claim 1, wherein, The training data of the scene recognition model is obtained by collecting operation steps performed by a plurality of in-service professional operators, a plurality of novice operators, and a plurality of students in the scene simulation mode.

10. A forklift driver simulation training system, characterized by, The method comprises a data acquisition module, a time sequence matching module, a scene recognition module and a prompt generation module, and the steps of the forklift driver simulation training method in any one of claims 1-9 are realized by cooperation of the data acquisition module, the time sequence matching module, the scene recognition module and the prompt generation module.

Images