An emergency evacuation VR agent modeling method based on Qwen2-VL multi-modal large model and application

Abstract

This invention discloses a VR intelligent agent modeling method and application for emergency evacuation based on Qwen2-VL. First, a VR emergency evacuation training scenario is constructed, and emergency evacuation tasks are designed based on this scenario. Real-person evacuation behavior data is collected to construct multimodal data including visual semantics, command behavior, and evacuation decision-making, creating a triplet dataset of "emergency command - evacuation scene image - evacuation action". The Qwen2-VL model is trained using this triplet dataset to construct an AI-NPC model with cross-modal emergency evacuation decision-making capabilities encompassing visual, linguistic, and action aspects. The trained AI-NPC model undergoes multi-stage emergency evacuation verification to validate its decision-making ability, scenario adaptability, and reliability. The intelligent NPC constructed using this method possesses cross-modal visual-linguistic-action fusion capabilities, can simulate the psychological state of real pedestrian evacuation decisions, and can quickly adapt to various emergency scenarios.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of virtual reality and artificial intelligence, specifically involving a VR scene intelligent agent modeling and control method for pedestrian emergency evacuation simulation, and in particular an emergency evacuation VR intelligent agent modeling method and application based on Qwen2-VL multimodal large model. Background Technology

[0002] With the acceleration of urbanization, the scale of densely populated places such as large public venues, transportation hubs, and commercial complexes is constantly expanding, making pedestrian emergency evacuation safety an increasingly prominent issue. In recent years, mass casualty accidents caused by sudden events such as fires, earthquakes, and terrorist attacks worldwide have commonly involved problems such as congested evacuation routes, personnel decision-making errors, and lack of guidance information. Conducting high-fidelity, interactive emergency evacuation simulation training has become one of the key means to improve the public's emergency response capabilities and optimize the safety design of venues.

[0003] In VR emergency evacuation simulations, non-player characters (NPCs) are the core element for simulating real pedestrian behavior and constructing group interaction scenarios. An ideal evacuation NPC should possess the following capabilities: the ability to perceive dynamic scene information such as fire sources, smoke, and obstacles; the ability to understand semantic prompts such as evacuation instructions and safety signs; the ability to adjust paths and behaviors in real time according to environmental changes; and the ability to exhibit real pedestrian psychological characteristics such as panic, conformity, and mutual assistance. However, existing VR emergency evacuation simulation systems generally suffer from low levels of intelligence in their NPCs. Most systems still use the A* algorithm to control NPC movement, only achieving simple obstacle avoidance and shortest path location, failing to simulate pedestrians' real-time perception and avoidance decisions regarding dynamic threats such as fire and smoke, and even less able to reflect the psychological and behavioral differences during evacuation. Furthermore, during real evacuation, pedestrians need to simultaneously process visual information (smoke color, exit signs), auditory information (alarm sounds, instructions), and environmental semantics (evacuation maps, indicator lights), while existing NPC models lack the ability to fuse and make decisions based on multi-source information, making it difficult to reproduce the patterns of crowd evacuation behavior in real-world scenarios.

[0004] Therefore, there is an urgent need for an intelligent NPC modeling method that has cross-modal fusion capabilities of vision, language, and action, can simulate the decision-making psychology of real pedestrians during evacuation, and can be quickly adapted to various emergency scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a modeling method and application for VR intelligent agents in emergency evacuation based on Qwen2-VL. Targeting VR pedestrian emergency evacuation simulation scenarios, it achieves end-to-end fusion of multimodal perception, semantic understanding, and behavioral decision-making of NPCs in dynamic evacuation environments, thereby improving their intelligence level, interaction realism, and scenario generalization ability in complex emergency scenarios.

[0006] To achieve the objectives of this invention, the following are embodiments of this invention:

[0007] A VR intelligent agent modeling method for emergency evacuation based on Qwen2-VL includes the following steps:

[0008] Step S1: Construct a VR emergency evacuation training scenario, design emergency evacuation tasks based on the scenario, collect real-person evacuation data to construct multimodal data including visual semantics, command behavior and evacuation decision, and construct a triplet dataset of "emergency command-evacuation scene image-evacuation action".

[0009] Step S2: The Qwen2-VL model is trained using the triplet dataset to build an AI-NPC model with cross-modal emergency evacuation decision-making capabilities of vision, language and action.

[0010] Step S3: Perform multi-stage emergency evacuation verification on the trained AI-NPC model to verify the model's decision-making ability, scenario adaptability, and reliability.

[0011] Furthermore, the basic VR scene construction and data collection mentioned in step S1 specifically includes the following steps:

[0012] Step S1.1: Construct a basic VR emergency evacuation training scenario;

[0013] Step S1.2, design emergency evacuation tasks, including but not limited to: (1) Single-target evacuation task: independently find the nearest safe exit according to the exit sign; (2) Command-guided evacuation task: select the designated exit according to the instructions of the virtual commander or emergency broadcast; (3) Dynamic threat avoidance task: sense the smoke / fire source ahead and actively detour, etc.

[0014] Step S1.3: In the constructed VR emergency scenario, real people complete typical emergency evacuation tasks and collect multimodal data during the evacuation process. The multimodal data includes, but is not limited to: VR head-mounted display image set of the subjects, eye-tracking data, and motion capture data.

[0015] Step S1.4: Perform timeline alignment, cleaning, and labeling on the collected data to form the following standardized subset datasets, including but not limited to the visual semantic dataset, the command behavior dataset, and the evacuation decision triplet dataset.

[0016] Furthermore, the steps in step S2 to construct the AI-NPC model are as follows:

[0017] Step S2.1: Construct the Qwen2-VL model structure, including a visual representation module, a language processing module, and a cross-modal fusion module, wherein:

[0018] (1) Visual representation module: The original VR scene images and video data are transformed into visual feature vectors that can interact with the language modality using a lightweight CNN backbone network, focusing on extracting key evacuation target features such as fire source, smoke, exit signs, and obstacles.

[0019] (2) Language processing module: The Transformer decoder architecture is used to encode external emergency commands into language feature vectors;

[0020] (3) Cross-modal fusion module: Deep fusion of visual features and language features is achieved through cross-modal attention mechanism.

[0021] Step S2.2: The “emergency command-evacuation scene image-evacuation action” triplet dataset constructed in step S1.3 is used as input to fine-tune the AI-NPC model, allowing the model to learn the reasoning process from visual information of the evacuation scene and emergency language commands to evacuation action decisions.

[0022] Furthermore, in the visual representation module, the feature map Fl of the l-th convolutional layer is generated through the following nonlinear transformation:

[0023]

[0024] In the formula, F l F represents the feature map output from layer l, used for subsequent semantic encoding; l−1 W represents the input feature map of layer l-1 (which is image I for the input layer); l Here are the weight parameters of the convolution kernel in layer l; * is the convolution operator, defined as [W*F](i, j) = ∑ m ∑ n W(m, n)⋅F(i+m , j+n;b l σ is the bias term; σ(⋅) is the ReLU activation function, defined as max(0, x).

[0025] Furthermore, each Transformer layer in the language processing module contains a feedforward network and residual connections:

[0026] ;

[0027] It also employs a residual connection and layer normalization structure:

[0028] ;

[0029] In the formula, FFN is a feedforward neural network, W1 and W2 are weight matrices, and b1 and b2 are bias terms. Sublayer(x) represents a multi-head attention sublayer or a feedforward network sublayer. This structural design ensures stable training of deep networks.

[0030] Furthermore, the cross-modal fusion module employs a 12-layer Transformer cross-attention layer, with the calculation process as follows:

[0031] 1. Scaling Dot Product Attention:

[0032] ;

[0033] 2. Bullish Attention:

[0034] ;

[0035] ;

[0036] In the formula, Q (Query) is the query vector, derived from the encoding of the emergency instruction text; K (Key) is the key vector, derived from the encoding of the visual feature map; V (Value) is the value vector, derived from the encoding of the visual feature map; d k It is the dimension of the key vector, used for scaling to prevent the gradient from vanishing due to an excessively large dot product; W is the linear projection parameter matrix of the i-th attention head; O is the output projection matrix; h is the number of attention heads. This mechanism generates a joint representation vector, achieving feature alignment between visual and linguistic modalities.

[0037] Furthermore, step S2.2 is optimized using the cross-entropy loss function combined with L2 regularization:

[0038]

[0039] Where N is the batch size, M is the number of action categories (9 categories), and y ij For realistic action tags, p ij To predict probabilities for the model, The regularization coefficient is . These are the model parameters. By minimizing this loss function, the model learns the inference mapping from visual information and language instructions to action decisions.

[0040] Furthermore, the AI-NPC model verification and evaluation in step S3 specifically includes the following steps:

[0041] Step S3.1, Basic evacuation function verification; The AI-NPC model is verified in a simple binary VR emergency evacuation scenario to check whether the model can complete basic evacuation decisions and path planning based on scene images and simple emergency instructions.

[0042] Step S3.2, evacuation scenario adaptability verification; The verified model is put into a newly built VR emergency evacuation scenario for verification. The newly built scenario is a simplified scenario and has differences in building layout, exit location and obstacle distribution compared with the original basic scenario built in step S1.1. The model's adaptability in non-training evacuation scenarios is tested to verify whether it can complete basic tasks such as personnel evacuation, fire source avoidance and exit location.

[0043] Step S3.3, Real-person Collaborative Evacuation Verification: Place the model and real players simultaneously in a VR emergency evacuation scenario for collaborative evacuation simulation verification. Compare the evacuation decision results of the model with the actions of real players in the same scenario and under the same emergency instructions. Evaluate the model from multiple dimensions and verify the reliability and behavioral realism of the AI-NPC model in actual emergency evacuation drill scenarios.

[0044] The AI-NPC model, constructed using the aforementioned Qwen2-VL-based VR intelligent agent modeling method for emergency evacuation, is applied to pedestrian emergency evacuation simulation. During the simulation, the AI-NPC model receives VR scene images and emergency commands, and then outputs discrete action decisions and continuous speed control signals.

[0045] ;

[0046] ;

[0047] Among them, A t+1 For discrete action decisions at time t+1. H is the joint representation at time t. t As a historical decision-making sequence, Choose the action 'a' that maximizes the probability. t+1 For continuous velocity control signals, α is the momentum coefficient used to smooth velocity changes, and W... v b v These are learnable parameters.

[0048] Compared with the prior art, the beneficial results of the present invention are:

[0049] (1) Significantly improved the intelligence level of NPCs in VR emergency evacuation scenarios.

[0050] This invention constructs a cross-modal fusion AI-NPC model based on the Qwen2-VL model, enabling the NPC to make flexible evacuation decisions based on dynamic threat visual information such as fire sources and smoke, as well as emergency broadcast instructions. This solves the problem that traditional A* algorithm NPCs can only perform fixed path planning and cannot perceive threats.

[0051] (2) It realizes visual-language-action cross-modal interaction for emergency evacuation.

[0052] This invention fine-tunes the model by collecting triplet data of "emergency instructions - evacuation scene images - evacuation actions", enabling the AI-NPC model to complete the reasoning mapping from evacuation scene images and emergency language instructions to the execution of evacuation actions. This solves the problem that NPCs in existing VR evacuation scenarios cannot understand humanized instructions such as "evacuate from the east exit".

[0053] (3) Possesses good adaptability to evacuation scenarios.

[0054] This invention designs a multi-stage evacuation verification process, from basic scenarios to differentiated new scenarios, and then to real-person collaborative verification, to ensure that the trained AI-NPC model can work stably in evacuation scenarios with different building structures and different disaster types, and has the reliability for actual emergency drill applications. Attached Figure Description

[0055] Figure 1 This is the overall flowchart of the present invention.

[0056] Figure 2 This is a schematic diagram of the multimodal data acquisition and AI-NPC model training architecture of the present invention.

[0057] Figure 3 This is a schematic diagram of the multi-stage verification process of the AI-NPC model of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, not all of them. Therefore, the following detailed description of the embodiments provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0060] like Figure 1 , Figure 2 and Figure 3 As shown, an emergency evacuation VR intelligent agent modeling method based on Qwen2-VL includes the following steps:

[0061] Step S1, VR emergency evacuation scenario construction and multimodal data acquisition, mainly includes the following steps:

[0062] Step S1.1, Construction of VR Emergency Evacuation Training Scenario

[0063] A VR emergency evacuation training scenario was built using the Unity 3D engine. The scenario is a 50m x 50m indoor space simulating a section of a large commercial complex. The scenario includes the following key elements:

[0064] (1) Evacuation routes: Set up a main route with a width of 2-4 meters and a secondary route with a width of 1.5 meters. Directional signs are set up at the intersections of the routes.

[0065] (2) Safety exit signs: Illuminated evacuation signs conforming to GB 13495 standard shall be installed at the end of the passage and at key turning points. The signs shall be 0.5 meters above the ground and visible at a distance of not less than 15 meters.

[0066] (3) Simulate fire source: Set up a particle system at random locations in the scene to simulate fire. The fire height is 0.5-2 meters and the fire color adopts a gradient rendering from RGB(255, 69, 0) to RGB(255, 140, 0).

[0067] (4) Smoke diffusion: The Volumetric Fog system of Unity is used to simulate smoke diffusion. The smoke concentration is set to 0.3-0.8 (range 0-1), the smoke diffusion speed is 0.2m / s, and the smoke color adopts the gradient from RGB(128, 128, 128) to RGB(64,64, 64).

[0068] (5) Obstacles: Randomly place obstacles such as tables, chairs, pillars, and guardrails, with a density of 3-5 obstacles per 100 square meters.

[0069] (6) Emergency broadcasting points: Virtual broadcasting sources are set up at the four corners of the scene, with a broadcasting coverage radius of 15 meters.

[0070] Step S1.2: Design three types of typical emergency evacuation tasks, with five different difficulty levels for each type of task.

[0071] (1) Single-target evacuation task: Subjects independently find the nearest safe exit based on exit signs. Exit signs are divided into two modes: solid green (normal state) and flashing green (emergency state). The difficulty of the task is adjusted by increasing the number of passage branches and changing the visibility of the signs.

[0072] (2) Command-guided evacuation task: Subjects select a designated exit according to the instructions from the virtual commander or emergency broadcast. The types of instructions include: "Please evacuate from the east exit", "There is heavy smoke ahead, please detour to the north passage", "Bend low and follow the green indicator light", etc. The instructions are played through the built-in headphones of the VR headset, and the voice is in female Mandarin, with a medium speaking speed (180 words per minute).

[0073] (3) Dynamic threat avoidance task: During the evacuation, the subjects sensed the smoke / fire source ahead and actively detoured around it. The fire source / smoke would randomly appear at the predetermined passage location 15 seconds after the start of the task, requiring the subjects to adjust their path in real time.

[0074] Step S1.3: Collect real-person evacuation behavior data. Fifty participants were recruited, aged 20-55 years, with a male-to-female ratio of 1:1, and none had a history of VR motion sickness. After signing an informed consent form, each participant underwent 5 minutes of VR environment adaptation training, followed by completing the three typical tasks mentioned above, with each task repeated 3 times.

[0075] The following multimodal data were collected simultaneously during the experiment:

[0076] (1) VR headset image acquisition: Using the HTC Vive Pro Eye headset, a first-person perspective image sequence was acquired at a resolution of 1920×1080 and a frame rate of 30fps. The image acquisition module recorded the visual attention area of ​​the subject for key targets such as fire sources, smoke, and exit signs.

[0077] (2) Eye-tracking data: Using the Tobii eye-tracking system built into the head-mounted display, with a sampling rate of 120Hz, the visual saccade paths and fixation hotspots of the subjects during the evacuation process were collected. Fixation point determination threshold: fixation duration ≥100ms, angle of view change ≤1°.

[0078] (3) Motion capture data: Using three HTC Vive trackers, respectively attached to the head, chest, and waist, the subject's three-dimensional spatial coordinates (x, y, z) and rotation angles (pitch, yaw, roll) were recorded at a sampling rate of 90Hz. Handle input data, including movement direction vector and grip state, were also recorded.

[0079] Step S1.4: The collected data is aligned on the time axis, cleaned and labeled to form a standardized subset dataset.

[0080] It includes a visual semantic dataset, a command behavior dataset, and an evacuation decision triplet dataset.

[0081] The visual semantic dataset was obtained by instance segmentation and semantic annotation of evacuation-related targets in image frames. The LabelMe annotation tool was used to create polygon annotations for safety exits, evacuation signs, fire sources, smoke, obstacles, and other pedestrians in each image frame. A total of 120,000 frames were annotated, with an average of 5.2 labeled targets per frame. The annotation format is COCO.

[0082] The instruction-action dataset was obtained by converting speech instructions into text and temporally associating them with the subjects' response actions. Response actions were defined as changes in movement direction and speed within 3 seconds of the instruction being read. A total of 8,500 valid instruction-action pairs were collected.

[0083] The evacuation decision triplet dataset is a standardized training sample set that constructs a one-to-one correspondence between "evacuation scene image - emergency instruction text - evacuation action label". The evacuation action labels include nine categories: {go straight, turn left, turn right, back, stop, bend over, accelerate, decelerate, crouch}. A total of 45,000 training samples were generated, divided into training, validation, and test sets in an 8:1:1 ratio.

[0084] Step S2, the construction and training of the AI-NPC model based on Qwen2-VL, includes the following steps:

[0085] Step S2.1: Build the Qwen2-VL model structure.

[0086] This embodiment uses Qwen2-VL as the base model. This model is a large multimodal model, comprising three core modules:

[0087] (1) Visual Representation Module: A lightweight CNN backbone network is used to convert the input VR image (resolution 224×224×3) into a visual feature vector. The visual encoder outputs a feature dimension of 1024. The feature map Fl of the l-th convolutional layer is generated through the following nonlinear transformation:

[0088]

[0089] Where * represents the convolution operator. For any spatial coordinate (i,j) on the output feature map, the expansion of this convolution operation is defined as:

[0090]

[0091] In the formula, (i,j) represents the spatial coordinate index of the output feature map; (m,n) represents the convolution kernel weight matrix W. l The traversal index (i.e., the width and height of the convolution kernel); W l (m,n) represents the weight value of the convolutional kernel of the l-th layer at position (m,n); F l−1 (i+m,j+n) is the input feature map F of the (l-1)th layer. l−1 The pixel value at the corresponding position in the middle; F l b represents the feature map output from layer l, used for subsequent semantic encoding; lThis is the bias term; σ(⋅) is the ReLU activation function, defined as σ(x)=max(0,x), where x is the weighted input value of each neuron calculated by the convolutional layer. If this value is less than 0, the output is 0; if it is greater than 0, the value is directly output. The module focuses on extracting features of key evacuation targets such as fire sources, smoke, exit signs, and obstacles, and enhances the weights of key regions through an attention mechanism.

[0092] (2) Language Processing Module: Responsible for parsing natural language instructions and semantic reasoning to obtain language features. A Transformer decoder architecture is adopted to encode external emergency instructions (such as "There is heavy smoke ahead, please bend over and move low") into 1024-dimensional language feature vectors. Each Transformer layer contains a feedforward network and residual connections:

[0093]

[0094] It also employs a residual connection and layer normalization structure:

[0095]

[0096] In this structure, FFN is a feedforward neural network, x is the input vector, W1 and W2 are weight matrices, b1 and b2 are bias terms, LayerNorm is a layer normalization operation used for stable training, Dropout is a random deactivation operation used to prevent overfitting, and Sublayer(x) represents a multi-head attention sublayer or a feedforward network sublayer. This structural design ensures stable training of deep networks.

[0097] (3) Cross-modal fusion module: Deep fusion of visual and linguistic features is achieved through a cross-modal attention mechanism. A 12-layer Transformer cross-attention layer is used, with each hidden layer having a dimension of 2048 and 16 multi-head attention heads. A joint representation vector with a dimension of 2048 is generated. The core calculation process is as follows:

[0098] 1. Scaling Dot Product Attention:

[0099]

[0100] 2. Bullish Attention:

[0101]

[0102] in,

[0103] In the formula, Q (Query) is the query vector, derived from the encoding of the emergency instruction text; K (Key) is the key vector, derived from the encoding of the visual feature map; V (Value) is the value vector, derived from the encoding of the visual feature map; d kThe dimension of the key vector is used for scaling to prevent the gradient from vanishing due to an excessively large dot product; softmax is the normalized exponential function that converts the input into a probability distribution; concat is the vector concatenation operation; head i This represents the i-th "attention head"; h is the number of attention heads; W is the linear projection parameter matrix of the i-th attention head; O This outputs the projection matrix. `Attention()` is a scaled dot product attention mechanism that generates a joint representation vector, achieving feature alignment between visual and linguistic modalities.

[0104] Step S2.2: The "emergency command-evacuation scene image-evacuation action" triplet dataset constructed in step S1.3 is used as input to the Qwen2-VL model constructed in step S2.1. The model is then optimized using the cross-entropy loss function combined with L2 regularization to obtain the AI-NPC model.

[0105]

[0106] in, The cross-entropy loss function is given by N, where N is the batch size and M is the number of action categories (9 categories). ij p represents the true label of the i-th sample in the j-th action category (1 if it belongs to that category, 0 otherwise); ij This represents the probability predicted by the model that the i-th sample belongs to the j-th action category. This is the regularization coefficient, used to prevent overfitting; For all learnable parameters of the model The L2 norm is used to minimize this loss function, enabling the model to learn the inference mapping from visual information and language instructions to action decisions.

[0107] After receiving VR scene images and emergency commands, the trained AI-NPC model outputs discrete action decisions and continuous speed control signals:

[0108]

[0109]

[0110] Among them, A t+1 For discrete action decisions at time t+1. It is a discrete action space (in this embodiment, it includes 9 types of actions such as going straight and turning left). H is the joint representation at time t. t As a historical decision-making sequence, Take the action 'a' that maximizes the probability; P(⋅) is the probability distribution of the model output; V t+1The signal is a continuous velocity control signal, where α is the momentum coefficient (0.3 in this embodiment) to ensure smooth motion. W v b v These are learnable parameters.

[0111] Step S3: Perform AI-NPC model validation and evaluation, such as... Figure 3 As shown, the specific steps include:

[0112] Step S3.1: Perform basic evacuation function verification. In a simple binary VR emergency evacuation scenario, perform basic verification on the AI-NPC model to check whether the model can complete basic evacuation decisions and path planning based on scene images and simple emergency instructions (such as "Evacuate from the left exit" or "There is a fire ahead, please detour").

[0113] Decision accuracy is used as the core evaluation indicator:

[0114]

[0115] Where, N correct For the model to correctly predict the number of actions, N total This represents the total number of tests. When Acc is greater than or equal to 85%, the model is considered to have passed the basic functional verification.

[0116] Step S3.2: Perform evacuation scenario adaptability verification. The verified model is then deployed into a newly created VR emergency evacuation scenario for further testing. This new scenario differs from the original basic scenario in building layout, exit locations, and obstacle distribution. This verifies the model's adaptability in non-trained evacuation scenarios. The accuracy reduction value is used to quantify the model's scenario adaptability.

[0117]

[0118] Among them Acc train Acc is the accuracy of the model in the training scenario. new This represents the model's accuracy in the new scenario. When... When the percentage is less than or equal to 15%, the model is considered to have good scene adaptability.

[0119] Step S3.3: Conduct real-person collaborative evacuation verification. Deploy the trained AI-NPC model in a VR system and conduct a comparative experiment with 30 real subjects in the same scene. At the same time, set up an NPC controlled by the traditional A* algorithm as a baseline control group to evaluate the reliability and practicality of the model from multiple dimensions.

[0120] More specifically, several NPCs controlled solely by the traditional A* algorithm are pre-set in the scene as a comparison reference. The number of NPCs is set to 20, their movement speed is 1.2m / s, and they only have the ability to avoid static obstacles.

[0121] The following indicators were used for quantitative evaluation:

[0122] (1) Efficiency improvement ratio: in, The average evacuation time for NPCs using the traditional A* algorithm. This represents the average evacuation time of the AI-NPC model in this invention.

[0123] (2) Time efficiency ratio: in, The evacuation completion time of the model. This represents the average evacuation time for real people.

[0124] (3) Path overlap: in, This is the cumulative length of the path that overlaps between the model and the real person (a distance of less than 0.5 meters is considered an overlap). This represents the total path length.

[0125] (4) Conflict frequency: in, The number of times the model and the real person had spatial conflicts (distance less than 0.5 meters). Total duration of coordinated evacuation (in seconds).

[0126] By referencing benchmark data from collaborative evacuation among real groups, a judgment value is set to determine whether the AI-NPC model possesses the reliability and behavioral realism of actual emergency evacuation drills.

[0127] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A Qwen2-VL-based emergency evacuation VR agent modeling method, characterized in that, Includes the following steps: Step S1: Construct a VR emergency evacuation training scenario, design emergency evacuation tasks based on the scenario, collect real-person evacuation data to construct multimodal data including visual semantics, command behavior and evacuation decision, and construct a triplet dataset of "emergency command - evacuation scene image - evacuation action". Step S2: The Qwen2-VL model is trained using the triplet dataset to build an AI-NPC model with cross-modal emergency evacuation decision-making capabilities of vision, language and action. Step S3: Perform multi-stage emergency evacuation verification on the trained AI-NPC model to verify the model's decision-making ability, scenario adaptability, and reliability.

2. The Qwen2-VL-based emergency evacuation VR agent modeling method according to claim 1, characterized in that, The basic VR scene construction and data acquisition mentioned in step S1 specifically includes the following steps: Step S1.1: Construct a basic VR emergency evacuation training scenario; Step S1.2: Design emergency evacuation tasks; Step S1.3: In the constructed VR emergency scenario, real people complete typical emergency evacuation tasks and collect multimodal data during the evacuation process. The multimodal data includes, but is not limited to: VR head-mounted display image set of the subjects, eye-tracking data, and motion capture data. Step S1.4: Perform timeline alignment, cleaning, and labeling on the collected data to form the following standardized subset datasets, including but not limited to the visual semantic dataset, the command behavior dataset, and the evacuation decision triplet dataset.

3. The Qwen2-VL-based emergency evacuation VR agent modeling method according to claim 2, characterized in that, Including but not limited to: (1) Single-target evacuation task: independently find the nearest safe exit according to the exit sign; (2) Command-guided evacuation task: select the designated exit according to the instructions of the virtual commander or emergency broadcast; (3) Dynamic threat avoidance task: sense the smoke / fire source ahead and actively detour.

4. The Qwen2-VL-based emergency evacuation VR agent modeling method according to claim 2, characterized in that, The steps for building the AI-NPC model in step S2 are as follows: Step S2.1: Construct the Qwen2-VL model structure, including a visual representation module, a language processing module, and a cross-modal fusion module, wherein: (1) Visual representation module: The original VR scene images and video data are transformed into visual feature vectors that can interact with the language modality using a lightweight CNN backbone network, focusing on extracting features of key evacuation targets such as fire source, smoke, exit signs, and obstacles; (2) Language processing module: The Transformer decoder architecture is used to encode external emergency commands into language feature vectors; (3) Cross-modal fusion module: Deep fusion of visual features and linguistic features is achieved through a cross-modal attention mechanism; Step S2.2: The "emergency command-evacuation scene image-evacuation action" triplet dataset constructed in step S1.3 is used as input to fine-tune the AI-NPC model, allowing the model to learn the reasoning process from visual information of the evacuation scene and emergency language commands to evacuation action decisions.

5. The Qwen2-VL-based emergency evacuation VR agent modeling method according to claim 4, characterized in that, Visual representation module: feature maps F of the first convolutional layer l generated by the following non-linear transformation: ； where F l denotes the output feature map of the l-th layer, which is used for the subsequent semantic encoding; F l−1 denotes the input feature map of the l-th layer; W l is the convolution kernel weight parameter of the l-th layer; * is the convolution operator, defined as [W*F](i,j)=∑ m ∑ n W(m,n)·F(i+m,j+n); b l is the bias term; σ(·) is the ReLU activation function, defined as max(0,x).

6. The emergency evacuation VR intelligent agent modeling method based on Qwen2-VL according to claim 4, characterized in that, Each Transformer layer in the language processing module contains a feedforward network and residual connections: ； It also employs a residual connection and layer normalization structure: ； In the formula, FFN is a feedforward neural network, W1 and W2 are weight matrices, b1 and b2 are bias terms, and Sublayer(x) represents a multi-head attention sublayer or a feedforward network sublayer. This structural design ensures stable training of deep networks.

7. The emergency evacuation VR intelligent agent modeling method based on Qwen2-VL according to claim 4, characterized in that, The cross-modal fusion module uses 12 Transformer cross-attention layers, and the calculation process is as follows:

1. Scaling Dot Product Attention: ； 2. Bullish Attention: ； ； In the formula, Q is the query vector, derived from the encoding of the emergency instruction text; K is the key vector, derived from the encoding of the visual feature map; V is the value vector, derived from the encoding of the visual feature map; d k It is the dimension of the key vector, used for scaling to prevent the gradient from vanishing due to an excessively large dot product; W is the linear projection parameter matrix of the i-th attention head; O is the output projection matrix; h is the number of attention heads; this mechanism generates a joint representation vector to achieve feature alignment between visual and linguistic modalities.

8. The emergency evacuation VR intelligent agent modeling method based on Qwen2-VL according to claim 4, characterized in that, Step S2.2 uses the cross-entropy loss function combined with L2 regularization for optimization: ； Where N is the batch size, M is the number of action categories, and y ij For realistic action tags, p ij To predict probabilities for the model, The regularization coefficient is . These are the model parameters; by minimizing this loss function, the model learns the reasoning mapping from visual information and language instructions to action decisions.

9. The emergency evacuation VR intelligent agent modeling method based on Qwen2-VL according to claim 4, characterized in that, The AI-NPC model verification and evaluation in step S3 specifically includes the following steps: Step S3.1, Basic evacuation function verification: The AI-NPC model is verified in a simple binary VR emergency evacuation scenario to check whether the model can complete basic evacuation decisions and path planning based on scene images and simple emergency instructions. Step S3.2, evacuation scenario adaptability verification; The verified model is put into a newly built VR emergency evacuation scenario for verification. The newly built scenario is a simplified scenario and has differences in building layout, exit location and obstacle distribution compared with the original basic scenario built in step S1.

1. The model's adaptability in non-training evacuation scenarios is tested to verify whether it can complete basic tasks such as personnel evacuation, fire source avoidance and exit location. Step S3.3, Real-person Collaborative Evacuation Verification: Place the model and real players simultaneously in a VR emergency evacuation scenario for collaborative evacuation simulation verification. Compare the evacuation decision results of the model with the actions of real players in the same scenario and under the same emergency instructions. Evaluate the model from multiple dimensions and verify the reliability and behavioral realism of the AI-NPC model in actual emergency evacuation drill scenarios.

10. Applying the intelligent agent constructed using the Qwen2-VL-based emergency evacuation VR intelligent agent modeling method described in claim 1 to pedestrian emergency evacuation simulation, characterized in that, During pedestrian emergency evacuation simulation, the AI-NPC model, after receiving VR scene images and emergency commands, outputs discrete action decisions and continuous speed control signals: ； ； Among them, A t+1 For discrete action decisions at time t+1. H is the joint representation at time t. t As a historical decision-making sequence, Choose the action 'a' that maximizes the probability; V t+1 For continuous velocity control signals, α is the momentum coefficient used to smooth velocity changes, and W... v b v These are learnable parameters.

Images