Virtual reality-based logistics warehouse picking training and action optimization method and system

Abstract

The application provides a logistics warehouse picking training and motion optimization method and system based on virtual reality, and relates to the technical field of intelligent warehousing. The method constructs a virtual warehouse environment linked with a database in real time and generates a picking task guide; collects depth images, inertial measurement data and physiological signals, and fuses to generate whole body posture data of a trainee; identifies non-standard actions based on the posture data and calculates work efficiency; combines the carrying weight, physiological state and motion data to perform ergonomics scoring, fatigue accumulation analysis and injury risk prediction; dynamically adjusts the task difficulty and controls the training process according to the training state; and finally generates a training evaluation report through motion data comparison. The application can improve the quality of logistics picking training and reduce the risk of work injury.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehousing technology, specifically to a method for logistics warehousing picking training and motion optimization based on virtual reality. Background Technology

[0002] Logistics warehousing picking operations involve repetitive movements such as frequent walking, bending over, and carrying. Traditional training relies on on-site teaching, which has drawbacks such as high venue costs, subjective motion evaluation, inability to provide real-time warnings of ergonomic risks, and incomplete data recording. Most existing VR training systems only simulate the visual environment and lack full-body motion capture, standardized recognition, and injury risk warning functions, and do not support personalized training and collaborative learning. Summary of the Invention

[0003] The purpose of this invention is to propose a method and system for logistics warehousing picking training and motion optimization based on virtual reality. The aim is to build a low-cost, configurable virtual warehousing environment; achieve accurate capture and occlusion compensation of full-body motion; identify non-standard movements in real time and analyze efficiency; provide ergonomic scoring and injury risk warning integrated with wearable devices; support personalized difficulty adaptation and multi-student collaborative training; and improve training quality and occupational health level through full-process guidance and digital twin comparison reports.

[0004] The present invention proposes a virtual reality-based logistics warehousing picking training and motion optimization method, which includes the following steps:

[0005] S1. Read the preset warehouse scene configuration file and standard model library, instantiate the shelf, goods and scanning equipment models; read the location records and bind them to the corresponding location models; read the initial task list and generate the visual guidance information for the first target location; S2. Simultaneously acquire depth images, color images, acceleration data, and angular velocity data. Extract two-dimensional coordinates of human joints based on depth images and convert them into three-dimensional coordinates of joints. In the case of joint occlusion, use acceleration data and angular velocity data to predict the position of occluded joints and form motion frames. S3. Receive the action frame and the target cargo location coordinates and operation information corresponding to the current task, call the standard action library to obtain the normal angle range of the corresponding joint, calculate the deviation between the actual angle of each joint in the action frame and the normal angle range, identify non-standard action labels, calculate the path efficiency based on the walking trajectory and identify repeated walking behavior, and output the efficiency score. S4. Calculate the ergonomic score based on the joint angle data in the action frame, calculate the cumulative fatigue index and injury probability based on the ergonomic score, heart rate data and electromyography signals, and generate real-time risk warning information. S5. Construct a task queue and state machine according to the preset process configuration file, continuously receive hand position, grasping state, scanning event, voice confirmation command and teacher control command, verify the state of the current task and execute state transition, update visual guidance information according to task execution results, and dynamically adjust training difficulty according to the efficiency score and standardization score. S6. Compare the joint angle data of each action frame in this training with the standard action library, and extract the joint data with the largest angle deviation to generate targeted training suggestions.

[0006] As a preferred embodiment, in step S1, the warehouse scene configuration file includes the number of aisles, the number of shelf rows, the number of shelf layers, and the coordinate origin parameter. The model in the standard model library is instantiated to the corresponding three-dimensional coordinate position according to the preset layout parameters. Step S1 also includes: starting an inventory information polling thread to periodically read the inventory information table, and when a change in the inventory quantity of a storage location is detected, performing operations such as hiding the product model, switching the material, generating a warning label, or restoring the product model on the corresponding storage location model.

[0007] As a preferred embodiment, step S2 includes: The depth image acquisition thread acquires depth and color images at a first preset frequency, and the inertial measurement unit acquisition thread acquires triaxial acceleration and angular velocity data at a second preset frequency. After establishing a synchronization reference based on the timestamp, the data is stored in a circular buffer. The Hourglass neural network is used to extract the two-dimensional pixel coordinates and corresponding confidence scores of each joint of the human body from the depth image, and the three-dimensional coordinates of the joints are generated by back-projection of the camera intrinsic parameter matrix and depth values. When the joint confidence is detected to be lower than a preset threshold, the position of the occluded joint is predicted by the rotation quaternion of the inertial measurement units of the head-mounted display and the controller, and then weighted and fused according to the visual estimation confidence and the inertial measurement unit confidence.

[0008] As a preferred embodiment, step S3 includes: Maintain the personalized offset matrix corresponding to the trainee, update the normal joint angle range in the basic standard action library by combining it with the personalized offset matrix, and perform the non-standard action recognition based on the updated normal angle range. Maintain walking trajectory data within a preset time window, calculate path efficiency based on the straight-line distance between the starting point and the target point and the actual walking distance, and match the actual trajectory with the theoretical shortest path through a dynamic time warping algorithm to identify repetitive walking behavior.

[0009] As a preferred embodiment, step S4 includes: Ergonomic scores are calculated based on the joint angle data of the torso, neck, upper arm, forearm, wrist, and leg in the motion frame, and load corrections are applied to the ergonomic scores in combination with the current weight of the goods being handled. The system collects ergonomic scores, number of bending-up cycles, average heart rate, and average electromyographic amplitude over a preset time period. It calculates fatigue increments based on preset weights, accumulates fatigue increments over all training cycles to generate a cumulative fatigue index, and calculates the probability of injury based on the cumulative fatigue index.

[0010] As a preferred embodiment, step S5 includes: The task queue is constructed according to the process configuration file, and the process index, task index, error count and process start timestamp are initialized. The state machine is used to verify the status and transition the process of going to the storage location, scanning the barcode, picking up the goods, handling and placing the goods. After completing a preset number of tasks, the training difficulty is dynamically adjusted based on the average efficiency score, standardization score, and error rate, and subsequent training tasks are generated based on the adjusted difficulty parameters. When a sudden event instruction or inventory abnormality information is received from the teacher's end, the current task execution status information is saved and pushed onto the status stack. A temporary branch task is inserted at the head of the task queue. After the branch task is completed, the original task execution status is restored and the original task process continues.

[0011] Furthermore, this invention discloses a virtual reality-based logistics warehousing picking training and motion optimization system, which includes: The virtual warehouse environment construction module is used to read the preset warehouse scene configuration file and standard model library, instantiate the models of shelves, goods and scanning equipment; read the location records and bind them to the corresponding location models; read the initial task list and generate visual guidance information for the first target location. The motion capture and full-body posture tracking module is used to collect depth images, color images, acceleration data and angular velocity data. Based on the depth images, it extracts the two-dimensional coordinates of human joints and converts them into three-dimensional coordinates. In the case of joint occlusion, it uses acceleration data and angular velocity data to predict the position of the occluded joint and form motion frames. The action standardization recognition module is used to receive action frames and the target cargo location coordinates and operation information corresponding to the current task, call the standard action library to obtain the normal angle range of the corresponding joints, calculate the deviation between the actual angle of each joint in the action frame and the normal angle range, identify non-standard action labels, calculate the path efficiency based on the walking trajectory and identify repeated walking behavior, and output an efficiency score. The ergonomics scoring and injury risk warning module calculates the ergonomics score based on joint angle data in motion frames, calculates the cumulative fatigue index and injury probability based on the ergonomics score, heart rate data and electromyography signals, and generates real-time risk warning information. The picking process configuration and guidance module is used to build a task queue and state machine, continuously receive hand position, grasping status, scanning events, voice confirmation instructions and teacher control instructions, verify the status of the current operation and execute state transition, update visual guidance information according to task execution results, and dynamically adjust training difficulty based on the efficiency score and standardization score. The training evaluation and report generation module compares the joint angle data of each action frame in this training with the standard action library, extracts the joint data with the largest angle deviation, and generates targeted training suggestions.

[0012] Furthermore, this invention discloses an electronic device comprising: a processor and a memory storing computer program instructions; wherein the processor, when executing the computer program instructions, implements the aforementioned virtual reality-based logistics warehousing picking training and motion optimization method.

[0013] Furthermore, this invention discloses a computer-readable storage medium storing at least one executable instruction, which, when executed on an electronic device, causes the electronic device to perform the aforementioned virtual reality-based logistics warehousing picking training and motion optimization method.

[0014] Compared with the prior art, the present invention has the following advantages: This method dynamically binds a low-cost, highly configurable virtual warehousing environment to an SQL database, supporting various warehouse layouts, picking modes, and emergency event injection, thus reducing investment in training hardware. It integrates a full-body motion capture solution using depth cameras and IMUs, combined with inverse kinematics occlusion compensation, solving the problem of tracking failure when limbs are occluded. A dynamically updated personalized standard movement library is established for each trainee, automatically identifying and correcting non-standard movements such as excessive twisting, excessive bending, and repetitive walking, overcoming the subjectivity inherent in traditional manual evaluation. This method integrates multi-source data such as joint angles, heart rate, and electromyography to construct a cumulative fatigue model and predict injury probability, effectively improving occupational health protection levels. The system supports intelligent difficulty adaptation and multi-trainee collaborative training, combining post-training comparisons with digital twins of standard models or the trainee's own historical best movements, along with comprehensive reports and targeted improvement suggestions covering multiple dimensions such as efficiency, ergonomics, and standardization, providing precise training feedback for trainees and instructors. This method stores the cumulative fatigue index and injury risk probability of each training session in the trainee's file, supporting long-term trend analysis and chronic injury early warning, achieving a complete closed loop from single training sessions to full-cycle occupational health management. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the overall architecture of the virtual reality-based logistics warehousing picking training and motion optimization system of this invention. Detailed Implementation

[0016] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0017] See Figure 1 As shown, this embodiment discloses a virtual reality-based logistics warehousing picking training and motion optimization system, including the following six key modules: virtual warehousing environment construction module, motion capture and full-body posture tracking module, motion standardization recognition module, ergonomic scoring and injury risk warning module, picking process configuration and guidance module, and training evaluation and report generation module. The overall system startup sequence is as follows: first, load the global configuration; then, start the virtual warehousing environment (including database connection); next, start the motion capture module and wait for the hardware to be ready; then, start the picking process guidance module; finally, start the remaining monitoring modules; and after all modules are ready, the VR training interface is displayed. The six modules are described in detail below.

[0018] I. Virtual Warehousing Environment Construction Module 1. Scene initialization: When the system starts, it reads the preset warehouse scene configuration file (number of aisles, number of rows of shelves, number of layers, coordinate origin, etc.) and the standard model library, and instantiates the models of shelves, goods, scanning equipment, etc. to the specified three-dimensional coordinate positions according to the layout parameters.

[0019] 2. Database Connection and Location Binding: Establish a persistent connection to the SQL database (automatic retry upon disconnection, 5-second interval, maximum 3 times). Retrieve all location records from the Location table (LocationID, 3D coordinates, current inventory). Maximum capacity Each location model can store information such as SKUID and product weight by adding a script component.

[0020] 3. Task Initialization and Visual Guidance: Read the initial task list (picking or sowing) from the Task table and determine the first highlighted target. Picking applies a highlight material to the target location and generates a floating arrow; sowing generates a semi-transparent light column and text label in each grid.

[0021] 4. Real-time inventory updates: A separate thread polls the Inventory table every second. When a specific storage location is detected... When changes occur, the model is updated: if it becomes 0, the product model is hidden and the material of the empty storage space is changed; if it is below the safety stock threshold (20% of the maximum capacity), a yellow warning sphere and a "low stock" label are displayed; if it changes from 0 to a positive number (replenishment), the model is restored and particle effects are played.

[0022] 5. Highlight switching after task completion: After the trainee completes the current task, the module receives the completion signal from the guidance module, updates the task status to "completed" in the database, obtains the LocationID of the next picking location, clears the old highlight and applies the new highlight and arrow.

[0023] 6. Emergency Injection: Receives teacher-side instructions via TCPSocket. Urgent Orders: Inserts a high-priority task at the head of the task queue; the corresponding storage location is highlighted in red and flashes twice as fast; a countdown timer is displayed on the task panel. Fire Alarm: Generates red smoke particles in the designated area and disables interactive ray detection. Event priority is as follows: Calculate (emergency order base score 10, fire alarm score 30), when Forced to be displayed on top.

[0024] 7. LOD Rendering Optimization: Before rendering each frame, the distance between the storage location model and the VR camera is calculated, and different detail levels are selected according to the distance (high-poly model within 10 meters, medium-poly model between 10 and 30 meters, low-poly model between 30 and 50 meters, and only outline beyond 50 meters). The threshold can be adjusted through the configuration file.

[0025] 8. Rendering Output: After all scenes are updated, the rendering pipeline is called to generate left and right eye views, which are then output to the VR headset via HDMI / DisplayPort. Then the process loops back to step 4.

[0026] 9. Collaborative Training Support: When the system is configured in collaborative mode, multiple students can access the same virtual warehouse through different VR devices. Each student has an independent virtual avatar, and the system maintains a global task list and dynamically assigns tasks to different students. The module detects whether two students enter the same narrow passage (distance < 1.5 meters). If a potential collision is detected, a semi-transparent red area is displayed in both students' fields of vision, and a voice prompt "Watch out for people ahead" is given. The teacher can view the real-time positions of all students and task progress.

[0027] II. Motion Capture and Full-Body Posture Tracking Module 1. Initialization: Load the depth camera intrinsic parameter matrix and distortion coefficients (use factory default values ​​and log warnings if calibration files are missing); initialize the IMU zero bias by statically placing 30 frames of data. Set the depth camera to 30fps and the IMU to 100fps, and establish a timestamp synchronization baseline.

[0028] 2. Parallel Data Acquisition: The depth image acquisition thread acquires one frame of depth map and color image every 1 / 30 of a second; the IMU acquisition thread reads the three-axis acceleration and angular velocity every 1 / 100 of a second. The most recent IMU timestamp is recorded for each depth map frame, and the data is temporarily stored in a circular buffer. If the depth map acquisition times out (>50ms), it is skipped and the dropped frame is recorded.

[0029] 3. 2D / 3D skeleton extraction: The Hourglass neural network is used to estimate the 2D pixel coordinates and confidence scores of 25 joints from the depth map. The 3D coordinates are obtained by back-projecting the intrinsic parameter matrix and depth values ​​onto the camera coordinate system, and then the joint angles are calculated.

[0030] 4. IMU-compensated occlusion: When the confidence level of a certain joint... At that time, the rotation quaternions of the head-mounted display and the controller IMU are used to predict the position of the occluded joints by combining human kinematic priors, and then weighted and fused with visual estimation (the visual confidence is weighted with a fixed IMU confidence of 0.7). If the visual confidence is lower than 0.1, the IMU prediction is used entirely.

[0031] 5. Full-body pose generation and filtering: Visual and IMU data are merged to obtain complete 3D coordinates of 25 joints, which are then converted into local Euler angles (roll / pitch / yaw for each joint) to form joint angle vectors. Simultaneously, the world coordinates of the torso center and the average position of both feet are calculated. A first-order low-pass filter (smoothing factor 0.4~0.6) is used to eliminate jitter.

[0032] 6. Sparse Node Pose Reconstruction (Headset + Controller Only): When the trainee is not wearing the motion capture suit, only three end points (head, hands, etc.) are known. Elbow, shoulder, and lower limb angles are estimated by inverse kinematics optimization (regularization coefficient 0.1, gradient descent iteration 10-20 times per frame). The lower limbs adopt a simplified "standing and walking" model (estimated based on trunk velocity and ground height).

[0033] 7. Wearable device data access: If the trainee wears a heart rate belt or electromyography armband, the module receives heart rate and electromyography signals via Bluetooth and adds the real-time heart rate value and average electromyography amplitude (representing the degree of muscle fatigue) to the motion frame for subsequent fatigue prediction.

[0034] 8. Pack and send motion frames: Each frame packs the timestamp, 3D coordinates and Euler angles of 25 joints, confidence of each joint, trunk center, ground projection velocity, hand grasping state, heart rate value, and electromyography amplitude into a motion frame, and sends it to other modules through shared memory.

[0035] 9. Loop: Clear the buffer, wait for the next frame of depth map interruption, and repeat steps 2-8. Maintain a stable 30fps, with end-to-end latency ≤20ms.

[0036] III. Action Standardization Recognition Module 1. Data reception: Read motion frames (joint angles, torso center, hand coordinates, grasping status) from shared memory, and obtain the target location coordinates for the current task from the picking process guidance module. and work process .

[0037] 2. Load Personalized Standard Library: Load the basic standard movement library (normal angle range and standard deviation of each joint in each segment) and the trainee's historical performance data from storage. The system maintains a personalized offset matrix for each trainee, initially zero, and gradually adjusts it according to the trainee's standardized movements during training. The normal range of each joint is updated as: basic range ± personalized offset (the upper limit of the offset is 0.5 times the basic standard deviation).

[0038] 3. Single-frame non-standard motion detection: Extract key quantities such as trunk bending angle, twisting angle, and hip, knee, and ankle angles. Calculate the deviation of the actual joint angles from the personalized normal range (marked if exceeding 2 standard deviations). Composite judgment: "Excessive twisting" is defined as lumbar twisting exceeding 15° and lateral hand displacement >0.3m; "Excessive bending" is defined as trunk bending >60° and knee bending <30°; "Knee bending <90° and ankle horizontal displacement >0.1m" is defined as knee going past the toes.

[0039] 4. Repeated Walking Detection: Maintain walking trajectories (sliding window) for the most recent 3 seconds and 30 seconds. Calculate path efficiency = straight-line distance from start to target / actual walking distance. If efficiency < 0.8, trigger DTW to compare the actual trajectory with the theoretical shortest path. If the DTW distance exceeds the threshold and there are round trips within the same area without picking, it is determined as repeated walking. When the redundant mileage rate exceeds 20%, a label is output.

[0040] 5. Efficiency index calculation: Real-time calculation of process time (starting from the current process), process efficiency ratio (standard time / actual time), scanning accuracy (number of correct scans / total number of scans), and idle walking ratio (percentage of time when hands are not grasping goods), and storage in the sliding buffer.

[0041] 6. Output: Output non-standard motion label vectors (motion type, timestamp, duration, severity) and efficiency score for each frame. It is sent to the ergonomics scoring module and the guidance module via a message queue.

[0042] 7. Personalized Standard Library Online Update: After each day's training, all "standard action" frames (unlabeled and efficiency > 80) of the trainee are summarized, clustered by stage, and the personalized offset is updated with a learning rate of 0.9 (smooth update). At the same time, the updated personalized standard library is added to the trainee's file for use in the next training session.

[0043] 8. Synchronization with the ergonomic module: The receiver maintains a 5-frame buffer, aligned based on timestamps, to avoid jitter.

[0044] IV. Ergonomic Scoring and Injury Risk Warning Module 1. Data Reception: Obtain joint angles, non-standard labels, efficiency scores, and trunk center for each frame from the motion standardization recognition module; load trainee personal parameters (age, gender, BMI, injury history) from the storage module; and call the interface provided by the guidance module. Obtain the total weight of the goods being transported; extract heart rate and electromyography amplitude from the motion frames.

[0045] 2. REBA Score Calculation: The angles of joints such as the trunk, neck, upper arm, forearm, and wrist are accurately measured using a three-dimensional planar normal. According to REBA standards: Group A (trunk, neck, legs) is scored based on angle mapping, plus load correction (add 0 for <5kg, add 1 for 5~10kg, add 2 for >10kg) to obtain the final score. Group B (upper arm, forearm, wrist) scores are based on angle mapping, plus grip correction. Final REBA score = ,in Check the REBA standard matrix. Points are awarded based on postural stability. Total scores range from 1 to 15, corresponding to risk levels (green / yellow / orange / red).

[0046] 3. Multimodal Cumulative Fatigue and Injury Risk Prediction: Each 5-second interval is considered a time period. The total number of seconds with a REBA score ≥ 8, the number of bending-up cycles, and the maximum REBA score are calculated within each interval. These are then accumulated using weights (0.3, 0.2, 0.2) to obtain the postural fatigue increment. Simultaneously, the average heart rate (the increase relative to resting heart rate) and average electromyographic amplitude (normalized to 0-1) within the same time period are calculated and added to the fatigue increment using weights (0.15, 0.15). The final fatigue increment formula for each time period is: The cumulative total over the entire training cycle is... Damage probability The age factor is calculated as min(1, max(0, (age-25) / 10)), and the BMI factor is similar. Output "High risk", >0.85 output "Extremely high risk, training is recommended to stop".

[0047] 4. Real-time Visual and Voice Feedback (merged with the original feedback module): Body Heatmap: The color of each joint's circular color block is mapped to that joint's contribution to REBA. The upper right panel displays the current REBA score, risk level, fatigue progress bar, and heart rate and EMG indicator lights (green for normal, yellow for high, red for danger). When an improper movement is detected, a diagram of the correct posture and a voice command (such as "Please bend your knees and keep your back straight") pop up in the center of the field of vision. When REBA is ≥8 for 3 consecutive frames or ≥11 for a single frame, the edge of the field of vision flashes red and a voice warning is repeated. If the training time exceeds 0.3 times the total training time, a rest suggestion will pop up. If the heart rate exceeds (220 - age) times 0.85 or the electromyographic amplitude is consistently higher than 0.8, a warning message "Physiological load is too high, please stop training" will pop up.

[0048] 5. Data Recording and Report Output: Records the REBA score, joint angles, and other data for each frame. , Average heart rate and average electromyography (EMG) data are written into the time-series database. At the end of training, average REBA, percentage of high-risk time, maximum consecutive high-risk duration, contribution of each body part, and periods of heart rate / EMG exceeding the standard are statistically analyzed, packaged into JSON, sent to the training evaluation module, and stored in the trainee's file.

[0049] 6. Loop: Wait for the next frame of data, repeat steps 1 to 5, and maintain 30fps.

[0050] V. Picking Process Configuration and Guidance Module 1. Load or Create Configuration: Check if a pre-set process configuration file (XML / JSON) exists, including picking mode, work sequence, time limit for each step, number of errors allowed, initial difficulty level, etc. If not, receive parameters through the graphical interface to generate a default configuration and save it. The difficulty level is divided into three levels: low, medium, and high, which affects the complexity of the task (such as the number of storage locations, time limit leniency, and branch task frequency).

[0051] 2. Construct the task queue and state machine: Harvesting-style task items: LocationID, SKUID, quantity, and collection point coordinates; Seeding-style: SlotID, SKUID, and expected quantity. Initialize state variables: stage index. Task Index Error count , Start timestamp of the process The virtual warehouse environment module is notified to generate initial visual guidance.

[0052] 3. Event Listening Loop: Continuously receives hand position and grasping status from the motion capture module, barcode scanning events from the virtual scanning device, voice confirmation commands, task panel buttons, sudden events from the teacher's end, and dynamic adjustment commands.

[0053] 4. Step verification and state transition (fruit-picking example): "Go to the storage location": The hand enters the target storage location interaction area and the grab status is "not grabbed" → Complete.

[0054] "Scan Barcode": Scan event SKUID matches task item → Correct, otherwise And a prompt.

[0055] "Pick up goods": The hand is in the interactive area and the grabbing status changes from "not grabbed" to "grabbed" → completed.

[0056] "Transportation": From the moment the goods are picked up, the hands enter the collection point area → completed.

[0057] "Placement": The hand is in the collection point area and the grabbing status changes from "grabbed" to "not grabbed" → Complete.

[0058] Seeding-style "placement" requires verification of SlotID matching; otherwise... And highlight the correct grid.

[0059] Each step is completed ;like If the total number of steps equals the total number of steps, then the current task is complete. Reset (Load the next task). If the number of failures exceeds the tolerance limit (default 3), the task is marked as "failed" and the reason for failure is recorded. It is then skipped or retried. Updates are performed after the state transition. It also notifies the environment module to update the visual guidance.

[0060] 5. Intelligent Dynamic Difficulty Adjustment: After every 5 tasks, the system evaluates the trainee's average efficiency score and standardization score. If the average efficiency score is >80, the standardization score is >70, and the error rate is <10%, the difficulty level is increased by one level (e.g., the number of storage locations increases by 20%, and the allowed time is reduced by 10%). If the average efficiency score is <50 and the error rate is >30%, the difficulty level is decreased by one level. The difficulty adjustment takes effect immediately, and subsequent tasks are generated according to the new difficulty. During adjustment, a prompt "Difficulty adjusted to level X" is displayed on the VR panel.

[0061] 6. Dynamic Guidance Update: The speech synthesis engine is invoked to generate guiding statements (such as "Please proceed to location A03"), and the volume and repetition count are dynamically adjusted based on the remaining time. For urgent orders, the visual color turns red and the flashing is doubled, and the voice adds "Urgent:".

[0062] 7. Branch Task Handling: Upon receiving a sudden event or inventory anomaly from a teacher: Save the current state (task index, stage index, error count, queue snapshot, time elapsed for each stage) and push it onto the state stack. Insert a temporary branch task (e.g., "go to the replenishment area to pick up goods") at the head of the task queue and set a branch marker. Pointing to this task, The original timer is paused, the guidance is overwritten and pointing to the branch target, and a voice prompt says "Please process the branch task first." After the branch task is completed, the save state pops up, the original queue and progress are restored, and the branch time is excluded from the original timer (the timer start point is moved to the next step).

[0063] When a student actively seeks help, the timer pauses, a text and image explanation pops up, and the time spent seeking help is not included in the standard time.

[0064] 8. Logs and Performance Metrics: Continuously record timestamps, task IDs, stage names, operation types, associated target IDs, and operation time. Calculate average stage time, error rate, task completion rate, and average response time (from teacher-injected event to student-initiated processing) in real time. Stages with an error rate exceeding 30% are marked as "difficult stages," and additional guidance elements are added to the request environment module.

[0065] 9. Training End: Send a notification when all tasks are completed / failed, or when the instructor forcibly ends the training, or when a student withdraws. The command clears the bootloader, packages the complete logs and sends them to the training and evaluation module, stores them in the history table, and resets the state machine to await the next startup.

[0066] VI. Training Evaluation and Report Generation Module 1. Real-time data caching: A receiving thread is started at the beginning of training, caching three types of data by timestamp through a message queue: Action standardization module: non-standard action labels, path efficiency, stage efficiency ratio, scan accuracy, and idle walking ratio for each frame. Ergonomics module: REBA score for each frame... , Joint angles, heart rate, and electromyography. Guidance module: Task completion log, branch task record, help request record, and difficulty adjustment record.

[0067] 2. Offline computation after training: Stop receiving data, align and clean the cached data by time, and calculate comprehensive metrics: Efficiency: Overall path efficiency (theoretical total length / actual total length), average stage efficiency ratio, overall accuracy, task completion rate, and average task time. Ergonomics: Average REBA, percentage of high-risk time (percentage of REBA ≥ 8 frames), maximum continuous high-risk duration, average REBA sub-score for each body part, and cumulative fatigue index. Final damage probability Total duration of heart rate / EMG exceeding normal limits. Standardization: Total number of non-standard movements and average duration, average redundant mileage rate of repeated walking.

[0068] 3. Overall score (out of 100):

[0069] in ; ; , The severity coefficient is calculated as follows: mild 0.3, moderate 0.6, severe 1.0. This represents the total number of frames.

[0070] 4. Digital Twin Motion Comparison and Replay: After training, trainees can select the "Motion Comparison" mode in the VR environment. The system dynamically times-aligns the joint angle data of all frames in this training session with the standard motion library (or the joint angles of the same segment in the trainee's best historical training), and generates a side-by-side dual-model animation (the left side is the standard / historical model, and the right side is the current model). Trainees can slow down, pause, and switch perspectives to observe the differences. The system marks joints with significant differences in red and displays the angle deviation value. A comparison report is also generated, listing the three joints and segments with the largest deviations and providing targeted practice suggestions.

[0071] 5. Generate a comprehensive report: PDF format, including training overview, efficiency analysis (curve / bar chart), ergonomic analysis (REBA time series, radar chart, fatigue progress bar, heart rate / EMG curves), statistics on non-standard movements (frequency bar chart, heat map, walking trajectory overlay), detailed records for each task, difficulty adjustment history, and personalized improvement suggestions (e.g., "Excessive bending over, it is recommended to practice knee squats; peak heart rate is too high, please pay attention to rest rhythm"). Raw data is exported as a CSV file (timestamps, 25 joint angles, REBA score, non-standard movement labels, heart rate, EMG), with a maximum single file size of 100MB; exceeding this size will result in multi-part compression.

[0072] 6. Storage and Notification: Reports are saved as "Student ID_Date and Time.pdf"; key values ​​are written to the database. surface; and Write into student file Table. A report summary is sent to the teacher's end via WebSocket, if it appeared during training. If the heart rate exceeds the standard for more than 5 minutes, a prominent reminder will be provided suggesting one-on-one counseling.

[0073] 7. Cleanup and Fault Tolerance: Clear the current cache and reset the internal state. If report generation fails, attempt to save to an alternative path and send an alert. Regularly clean up expired data according to the retention policy.

[0074] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for logistics warehousing picking training and motion optimization based on virtual reality, characterized in that, Includes the following steps: S1. Read the preset warehouse scene configuration file and standard model library, instantiate the shelf, goods and scanning equipment models; read the location records and bind them to the corresponding location models; read the initial task list and generate the visual guidance information for the first target location; S2. Simultaneously acquire depth images, color images, acceleration data, and angular velocity data. Extract two-dimensional coordinates of human joints based on depth images and convert them into three-dimensional coordinates of joints. In the case of joint occlusion, use acceleration data and angular velocity data to predict the position of occluded joints and form motion frames. S3. Receive the action frame and the target cargo location coordinates and operation information corresponding to the current task, call the standard action library to obtain the normal angle range of the corresponding joint, calculate the deviation between the actual angle of each joint in the action frame and the normal angle range, identify non-standard action labels, calculate the path efficiency based on the walking trajectory and identify repeated walking behavior, and output the efficiency score. S4. Calculate the ergonomic score based on the joint angle data in the action frame, calculate the cumulative fatigue index and injury probability based on the ergonomic score, heart rate data and electromyography signals, and generate real-time risk warning information. S5. Construct a task queue and state machine according to the preset process configuration file, continuously receive hand position, grasping state, scanning event, voice confirmation command and teacher control command, verify the state of the current task and execute state transition, update visual guidance information according to task execution results, and dynamically adjust training difficulty according to the efficiency score and standardization score. S6. Compare the joint angle data of each action frame in this training with the standard action library, and extract the joint data with the largest angle deviation to generate targeted training suggestions.

2. The method for logistics warehousing picking training and motion optimization based on virtual reality according to claim 1, characterized in that, In step S1, the warehouse scene configuration file includes the number of aisles, the number of rows of shelves, the number of shelf layers, and the coordinate origin parameter. The model in the standard model library is instantiated to the corresponding three-dimensional coordinate position according to the preset layout parameters. Step S1 also includes: starting an inventory information polling thread to periodically read the inventory information table, and when a change in the inventory quantity of a storage location is detected, performing operations such as hiding the product model, switching the material, generating a warning label, or restoring the product model on the corresponding storage location model.

3. The method for logistics warehousing picking training and motion optimization based on virtual reality according to claim 1, characterized in that, Step S2 includes: The depth image acquisition thread acquires depth and color images at a first preset frequency, and the inertial measurement unit acquisition thread acquires triaxial acceleration and angular velocity data at a second preset frequency. After establishing a synchronization reference based on the timestamp, the data is stored in a circular buffer. The Hourglass neural network is used to extract the two-dimensional pixel coordinates and corresponding confidence scores of each joint of the human body from the depth image, and the three-dimensional coordinates of the joints are generated by back-projection of the camera intrinsic parameter matrix and depth values. When the joint confidence is detected to be lower than a preset threshold, the position of the occluded joint is predicted by the rotation quaternion of the inertial measurement units of the head-mounted display and the controller, and then weighted and fused according to the visual estimation confidence and the inertial measurement unit confidence.

4. The method for logistics warehousing picking training and motion optimization based on virtual reality according to claim 1, characterized in that, Step S3 includes: Maintain the personalized offset matrix corresponding to the trainee, update the normal joint angle range in the basic standard action library by combining it with the personalized offset matrix, and perform the non-standard action recognition based on the updated normal angle range. Maintain walking trajectory data within a preset time window, calculate path efficiency based on the straight-line distance between the starting point and the target point and the actual walking distance, and match the actual trajectory with the theoretical shortest path through a dynamic time warping algorithm to identify repetitive walking behavior.

5. The method for logistics warehousing picking training and motion optimization based on virtual reality according to claim 1, characterized in that, Step S4 includes: Ergonomic scores are calculated based on the joint angle data of the torso, neck, upper arm, forearm, wrist, and leg in the motion frame, and load corrections are applied to the ergonomic scores in combination with the current weight of the goods being handled. The system collects ergonomic scores, number of bending-up cycles, average heart rate, and average electromyographic amplitude over a preset time period. It calculates fatigue increments based on preset weights, accumulates fatigue increments over all training cycles to generate a cumulative fatigue index, and calculates the probability of injury based on the cumulative fatigue index.

6. The method for logistics warehousing picking training and motion optimization based on virtual reality according to claim 1, characterized in that, Step S5 includes: The task queue is constructed according to the process configuration file, and the process index, task index, error count and process start timestamp are initialized. The state machine is used to verify the status and transition the process of going to the storage location, scanning the barcode, picking up the goods, handling and placing the goods. After completing a preset number of tasks, the training difficulty is dynamically adjusted based on the average efficiency score, standardization score, and error rate, and subsequent training tasks are generated based on the adjusted difficulty parameters. When a sudden event instruction or inventory abnormality information is received from the teacher's end, the current task execution status information is saved and pushed onto the status stack. A temporary branch task is inserted at the head of the task queue. After the branch task is completed, the original task execution status is restored and the original task process continues.

7. A virtual reality-based logistics warehousing picking training and motion optimization system, characterized in that, include: The virtual warehouse environment construction module is used to read the preset warehouse scene configuration file and standard model library, instantiate the models of shelves, goods and scanning equipment; read the location records and bind them to the corresponding location models; read the initial task list and generate visual guidance information for the first target location. The motion capture and full-body posture tracking module is used to collect depth images, color images, acceleration data and angular velocity data. Based on the depth images, it extracts the two-dimensional coordinates of human joints and converts them into three-dimensional coordinates. In the case of joint occlusion, it uses acceleration data and angular velocity data to predict the position of the occluded joint and form motion frames. The action standardization recognition module is used to receive action frames and the target cargo location coordinates and operation information corresponding to the current task, call the standard action library to obtain the normal angle range of the corresponding joints, calculate the deviation between the actual angle of each joint in the action frame and the normal angle range, identify non-standard action labels, calculate the path efficiency based on the walking trajectory and identify repeated walking behavior, and output an efficiency score. The ergonomics scoring and injury risk warning module calculates the ergonomics score based on joint angle data in motion frames, calculates the cumulative fatigue index and injury probability based on the ergonomics score, heart rate data and electromyography signals, and generates real-time risk warning information. The picking process configuration and guidance module is used to build a task queue and state machine, continuously receive hand position, grasping status, scanning events, voice confirmation instructions and teacher control instructions, verify the status of the current operation and execute state transition, update visual guidance information according to task execution results, and dynamically adjust training difficulty based on the efficiency score and standardization score. The training evaluation and report generation module compares the joint angle data of each action frame in this training with the standard action library, extracts the joint data with the largest angle deviation, and generates targeted training suggestions.

8. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements a virtual reality-based logistics warehousing picking training and motion optimization method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on an electronic device, causes the electronic device to perform a virtual reality-based logistics warehousing picking training and motion optimization method as described in any one of claims 1 to 6.

Images