VR scene intelligent identification method based on AI

Abstract

The invention belongs to the field of data processing, and discloses an AI-based VR scene intelligent identification method, which comprises the following steps: step 1, when a user uses a VR head-mounted display, obtaining an image frame I, a head posture H and a spatial audio A displayed by the VR head-mounted display, processing I, H and A to obtain a time alignment triple sequence, the time alignment triple comprising the image frame, the head posture and the audio direction; step 2, respectively acquiring image features of each image frame, and acquiring fused semantic features of each time point based on the image features; 3, constructing a spatial semantic map based on the fused semantic features and the audio direction; step 4, performing graph propagation operation on the nodes in the spatial semantic graph to obtain embedded representation of each node in the spatial semantic graph; step 5, acquiring a response node set based on the embedded representation; and step 6, updating the embedded representation of the nodes in the spatial semantic map based on the response node set. According to the invention, the complete understanding capability of the scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an AI-based VR scene intelligent recognition method. Background Art

[0002] With the widespread application of virtual reality (VR) technology in fields such as education, healthcare, gaming, and industrial simulation, users are increasingly demanding real-time interaction with virtual content in immersive environments. To enhance the intelligence of VR systems, researchers are exploring the use of artificial intelligence (AI) technologies to achieve intelligent understanding and automatic recognition of VR scenes. The core goal of this approach is to enable systems to identify the user's virtual scene, including the environment type, semantic objects, spatial structure, and interaction events, thereby supporting advanced features such as intelligent prompts, automatic navigation, and behavior prediction. Existing technologies primarily focus on two areas: image-based object detection and classification methods, which extract and recognize features from image frames in the user's current field of view to determine object or environment labels within the scene; and behavioral inference methods based on speech or interaction logs, which analyze user voice input and action sequences to generate simple semantic scene inferences. While these methods have achieved some success in static or low-complexity scenes, they still face limitations in complex, highly dynamic, and spatially rich VR environments.

[0003] First, traditional image recognition models are mostly based on single-frame image input and lack the ability to model the user's continuous perspective changes and gaze trajectory. This makes it difficult to achieve temporally consistent semantic understanding, resulting in semantic jumps and poor stability when the user quickly turns their head, moves, or traverses multiple areas. Second, existing methods often ignore or fail to fully utilize the highly structured three-dimensional spatial information in VR, such as scene boundaries and spatial relationships between objects. This makes it difficult for the system to identify spatial semantic structures (e.g., "the exit is behind the corner") or the paths of dynamic events. Third, although some studies have incorporated speech or audio data as auxiliary methods, these are mostly limited to content semantic recognition (e.g., speech-to-text) and do not leverage the unique spatial audio characteristics of VR for direction perception and event localization. Consequently, they are unable to identify key aspects such as "where the sound comes from" and "which scene change the sound corresponds to." Overall, current technology lacks a unified recognition system that can integrate temporal, spatial, and semantic features, making it difficult to meet the high-level intelligent recognition requirements of immersive, multimodal, and dynamic VR scenes. Summary of the Invention

[0004] The purpose of the present invention is to disclose an AI-based VR scene intelligent recognition method to solve the technical problems pointed out in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides an AI-based VR scene intelligent recognition method, comprising:

[0007] Step 1: When a user uses a VR headset, obtain the image frame I, head posture H, and spatial audio A displayed by the VR headset, process I, H, and A to obtain a sequence of time-aligned triplets, where the time-aligned triplets include the image frame, head posture, and audio direction;

[0008] Step 2: Obtain the image features of each image frame respectively, and obtain the fused semantic features of each time point based on the image features;

[0009] Step 3: Construct a spatial semantic map based on the fusion semantic features and audio direction;

[0010] Step 4: Perform graph propagation operations on the nodes in the spatial semantic graph to obtain the embedded representation of each node in the spatial semantic graph;

[0011] Step 5: Obtain the response node set based on the embedded representation;

[0012] Step 6: Update the embedding representation of the nodes in the spatial semantic graph based on the response node set.

[0013] Furthermore, the head posture H is measured by the IMU of the VR headset; the image frame I is obtained from the video channel of the VR headset.

[0014] Furthermore, I, H, and A are processed to obtain a time-aligned triple sequence, including:

[0015] The image frame obtained at time point t is represented as I t , for the head posture H obtained at time point t' t′ Perform linear interpolation resampling to calculate the head posture H at time point t t ;

[0016] Perform weighted averaging of the direction vectors of all spatial audio within the time window centered at time point t and with a window half-width of δ to obtain the global audio direction Set the global audio direction Project to the local coordinate system to obtain the final normalized audio direction

[0017] The time-aligned triplet at time point t is The triple sequence is represented as T represents the length of the time series of continuous processing, and t0 represents the starting time point of the time series of continuous processing.

[0018] Furthermore, step 2 includes:

[0019] From the image frame I t Extract the visual feature vector F t ;

[0020] Set the head posture H t Converted to a three-dimensional gaze direction unit vector hc t ;

[0021] The fused semantic features of time point p are represented as v p .

[0022] Furthermore, step 3 includes:

[0023] Combined I t The depth information carried by the object being watched in the image frame is back-projected into three-dimensional space to obtain the spatial position of the object being watched

[0024] The fused semantic feature v at time point t t and p t As the node construction element, define the initial node set V of the spatial semantic graph t ;

[0025] Get potential event nodes based on audio direction

[0026] Based on V t and Constructing spatial semantic graph G t .

[0027] Furthermore, the edge set E of the spatial semantic graph t The calculation process of the edge weight in includes:

[0028] The weight of the edge is calculated based on the spatial position, semantic features and gaze weight of the two nodes corresponding to the edge.

[0029] Furthermore, potential event nodes are obtained based on the audio direction include:

[0030] Based on audio direction The user's current spatial position p u As the starting point, construct an assumed sound source position r is the preset distance constant, Generate potential event nodes

[0031] Furthermore, step 4 includes:

[0032] During the graph propagation process, the edge e ij The information propagation introduces the type weight correction term ψ ij ,

[0033] Based on ψ ij Perform node updates.

[0034] Furthermore, step 5 includes:

[0035] The embedding activation level Act(h i ) to construct candidate regions for indicators

[0036] Act(h i ) is higher than the set threshold τ a The nodes are classified into candidate regions

[0037] For each node i in the candidate area, the audio direction driven priority weighting factor ω is further introduced i ;

[0038] ω i Sort from large to small and take the top-K nodes as the response node set elements, and for each node Assign response level L i , L i Used to decide what response behavior the system should take.

[0039] Furthermore, step 6 includes:

[0040] Calculate the consistency discrimination value of each node in the response node set respectively;

[0041] The embedding representation of the node is updated based on the consistency discriminant value.

[0042] Beneficial effects:

[0043] The present invention proposes an intelligent scene recognition method for VR environment, which systematically introduces a temporal perception mechanism, a spatial perception modeling mechanism and a multimodal fusion mechanism, fundamentally improving the system's ability to fully understand the scene. The present invention is based on the user's dynamic perception process in the VR environment, and uses the user's continuous perspective behavior to construct a temporal semantic chain to improve the coherence and context consistency of recognition; at the same time, based on the three-dimensional structure and spatial boundaries of the scene, a spatial semantic map is established, so that the system can understand the spatial layout and target relationship, thereby identifying high-order semantic content such as regional attributes, traffic paths, spatial events, etc.; in addition, by introducing the sound source direction information from the spatial audio channel, the present invention constructs a recognition mechanism for multimodal events, realizing the perception completion and semantic aggregation of invisible events. The above mechanisms are unified and integrated into a scene recognition framework, which can adapt to the user's behavior path, perception blind spots and environmental changes, and truly realize the dynamic, complete and semantic recognition of VR scenes, significantly improving the stability, accuracy and environmental adaptability of the recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a schematic diagram of an AI-based VR scene intelligent recognition method of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0047] like Figure 1 In one embodiment shown, the present invention provides an AI-based VR scene intelligent recognition method, comprising:

[0048] Step 1: When a user uses a VR headset, obtain the image frame I, head posture H, and spatial audio A displayed by the VR headset, process I, H, and A to obtain a sequence of time-aligned triplets, where the time-aligned triplets include the image frame, head posture, and audio direction;

[0049] Step 2: Obtain the image features of each image frame respectively, and obtain the fused semantic features of each time point based on the image features;

[0050] Step 3: Construct a spatial semantic map based on the fusion semantic features and audio direction;

[0051] Step 4: Perform graph propagation operations on the nodes in the spatial semantic graph to obtain the embedded representation of each node in the spatial semantic graph;

[0052] Step 5: Obtain the response node set based on the embedded representation;

[0053] Step 6: Update the embedding representation of the nodes in the spatial semantic graph based on the response node set.

[0054] Furthermore, the head pose H is measured by the VR headset's IMU (typically with a sampling frequency of 100 Hz). The image frame I is obtained from the VR headset's video channel, with a default frame rate of 30 Hz and bound to a precise time point t to obtain a gaze direction that is precisely aligned with the image frame.

[0055] Furthermore, I, H, and A are processed to obtain a time-aligned triple sequence, including:

[0056] The image frame obtained at time point t is represented as I t , for the head posture H obtained at time point t′ t′ Perform linear interpolation resampling to calculate the head posture H at time point t t ;H t It contains three Euler angles (pitch, yaw, and roll), corresponding to the user's pitch, horizontal rotation, and roll angles. The interpolation process is based on the image time points to build a cross-modal time synchronization structure, avoiding inconsistencies in the original modal time alignment.

[0057] Compared with vision and head movement, spatial audio t′ =(θ t′ ,φ t′ ,a t′ ) has a much higher sampling frequency (e.g. 24kHz), which represents the azimuth, elevation and sound pressure level of the current sound source. Sampling directly at the image frame time point will introduce high-frequency jitter. Therefore, the present invention performs a weighted average of the direction vectors of all spatial audio within a time window (i.e. [t-δ, t+δ]) centered at time point t and with a window half-width of δ to obtain the global audio direction.

[0058] Weighting is to treat each sound source direction in the time window as a three-dimensional direction vector, use its corresponding sound pressure level as the weight, perform weighted summation and then normalize it to obtain a unit vector representing the overall dominant sound source direction in the time period.

[0059] Set the global audio direction Project to the local coordinate system to obtain the final normalized audio direction

[0060]

[0061] R(H t ) represents the head posture H of the user at time t t The constructed three-dimensional rotation matrix (converting Euler angles into coordinate transformation matrix), δ can be set to 100ms; It can be used as a basis for spatial positioning of sound source events.

[0062] In addition, to cope with the short-term data loss that is common in VR devices, such as image frame loss, IMU delay or audio silence, the present invention builds a real-time compensation mechanism for modality loss at the data acquisition end. t When the audio is missing, the image direction of the middle frame is estimated based on the image information of the adjacent frames and the head motion trajectory; when the audio is missing, the direction of the previous frame audio is kept unchanged or a longer time window is used for interpolation to maintain x t The integrity and stability of the data structure;

[0063] The time-aligned triplet at time point t is The triple sequence is represented as T represents the length of the time series of continuous processing, and t0 represents the starting time point of the time series of continuous processing.

[0064] x t It is strictly aligned with the image frames in time and normalizes the user's perspective in space, becoming a unified semantic input structure for subsequent multimodal modeling and graph construction.

[0065] In a virtual reality environment, the user's immersive experience comes from multiple stimuli of images, sounds, and body sensations. For the system, it is necessary to collect information from multiple channels such as images, audio, and inertial sensors. However, these modal data often come from different sensors with different sampling frequencies and transmission delays. If they are not processed uniformly, it is very easy to cause temporal dislocation and spatial misperception in modeling, especially when the user turns his head quickly or switches his gaze object quickly. The semantic inconsistency between "what the user sees" and "what the actual user feels" perceived by the model is easy to occur. For this reason, in step 1 of the present invention, a three-modal alignment and spatial mapping fusion method with image frames as the time reference is proposed to uniformly organize image frames I t , head posture H t With spatial audio Finally, a standard multimodal input structure is generated It serves as the basis for subsequent semantic modeling and spatial map construction.

[0066] Furthermore, step 2 includes:

[0067] From the image frame I t Extract the visual feature vector F t , F t =f enc (I t ), where f enc (·) is a lightweight visual encoding module (such as a 3-layer convolution + pooling structure). The spatial dimension of the visual feature vector remains fixed to ensure the consistency of subsequent sequence splicing and operations;

[0068] Set the head posture H t Converted to a three-dimensional gaze direction unit vector hc t ;

[0069] Construct the inter-frame gaze distance matrix D, and the calculation formula of the elements in D is:

[0070] D p,q =||hc p -hc q ||2

[0071] hc p and hc q are the three-dimensional gaze direction unit vectors at time points p and q respectively; for example, when yaw = 45° and pitch = 0°, we can get D p,q The larger the value, the greater the difference in the user’s gaze direction at two time points. p,q represents the inter-frame gaze distance between time points p and q; hc p and hc q The head posture H at time point p is respectively p and the head pose H at time point qq The Euler angles are obtained by spherical coordinate mapping;

[0072] Construct a dynamic temporal weight matrix based on gaze differences. The calculation formula of the elements in the dynamic temporal weight matrix is:

[0073]

[0074] In the above formula, the first term is the Softmax normalization of the gaze difference, and the second term is the volatility penalty term of the gaze path;

[0075] α p,q represents the dynamic temporal weights at time points p and q, γ is the temperature coefficient that controls the sensitivity to gaze differences. For example, setting γ = 5 ensures that the weights are sensitive to small changes in viewing angle; Var(hc p-n:p ) represents the variance of the three-dimensional gaze direction unit vector from time point pn to p, which is used to penalize the risk of misaggregation caused by rapid viewpoint jitter; λ is a coefficient that controls the degree of regularization (e.g., λ = 0.1). D p,r =||hc p -hc r ||2,hc r The unit vector representing the three-dimensional gaze direction at time point r.

[0076] This innovative penalty term is specially designed to address the "micro-motion instability" phenomenon of VR headsets. The weight is almost unaffected in static observation, but it can effectively suppress the feature participation of abnormal frames in the rapid scanning stage.

[0077] After obtaining the dynamic temporal weight matrix, the present invention performs weighted coding on the visual features of consecutive frames to generate a semantic representation of the context of the current frame, and represents the fused semantic features of the time point p as v p ,

[0078]

[0079] F q Represents the visual feature vector of the image frame at time point q, α p,q Represents the dynamic temporal weights of time points p and q.

[0080] v p It represents the user's overall cognitive expression in the context of the current gaze behavior, and has temporal continuity and directional consistency.

[0081] The entire aggregation process is to build a local perception window centered on the current frame, and its size n can be dynamically adjusted. For example, it is set to n = 4 in static interaction scenarios and n = 2 in fast perspective exploration scenarios to reduce modeling delay.

[0082] Step 2 combines the time-aligned triplet sequence output from step 1 with the image content I t and head posture H t , extracting the semantic representation sequence formed in the user's continuous observation path, so as to more accurately reflect the user's "what he is looking at", "what he is looking at" and "what he has just looked at". In VR scenarios, this kind of sequence modeling is extremely important: on the one hand, the VR user's gaze often has clear intentions and goals; on the other hand, the gaze object may be partially visible in consecutive frames due to occlusion, rotation or displacement, so its complete semantics must be modeled using consecutive frames. In addition, the gaze association between image frames in the VR environment is not linearly time-dependent, but is closely related to the "viewpoint rotation path", which poses a challenge to traditional temporal modeling. Therefore, in this step, a temporal weight mechanism is constructed by modeling the difference in user gaze vectors, and a regular inhibition term for "attention instability" is introduced in feature aggregation to ensure that the semantic representation not only reflects historical observation behavior but also has a certain degree of robustness to sudden changes.

[0083] Sequences that integrate semantic features can accurately express the user's "target of attention" and "in what context" in the VR scene.

[0084] Through modeling in this step, the image from step 1 is organically combined with the head direction information to generate stable semantic input that can be used for graph construction and event reasoning. This solves the problem of "static frame discontinuity, target drift, and perspective mutation" in traditional recognition, which makes it difficult to stably understand user intentions, and enhances the system's structural understanding of user behavior.

[0085] Furthermore, step 3 includes:

[0086] Combined I t The depth information carried (generated by the VR engine or RGBD image) is used to back-project the target being watched in the image frame (such as the area around the pixel with the smallest depth) into three-dimensional space to obtain the spatial position of the target being watched

[0087] The fusion semantic features at time point t and p t As the node construction element, define the initial node set V of the spatial semantic graph t , where each node i contains two main attributes: fusion semantic features With spatial position p i , and an attention weight s i , used to record the user's gaze concentration;

[0088] The edge set E of the spatial semantic graph t The calculation process of the edge weight in includes:

[0089] The weight of the edge is calculated based on the spatial position, semantic features and gaze weight of the two nodes corresponding to the edge.

[0090] Specifically, the edge weight is calculated based on the spatial position, semantic features, and gaze weight of the two nodes corresponding to the edge, including:

[0091] For the edge set E of the semantic graph t The edge between nodes i and j is e ij The weight φ ij is defined as:

[0092]

[0093] in:

[0094] p i and p j Represent the spatial positions of nodes i and j respectively;

[0095] represents the fused semantic features of node i; is the fused semantic feature of node j;

[0096] s i and s j are the attention weights of nodes i and j respectively;

[0097] s i and s j Use f to transform the three-dimensional gaze direction unit vector hc i and hc j The difference between the mean of all three-dimensional gaze direction unit vectors within the set time window is input into the softmax function to obtain the normalized weight value;

[0098] For example, the set time window may be [t-δ1, t+δ1], where δ1 is the set duration, for example, 100 ms.

[0099] β1 and β2 are the spatial and semantic weight parameters, respectively. For example, β1 and β2 can be set to 0.6 and 0.4, respectively.

[0100] λ1 is the gaze co-amplification factor, which is used to enhance the edge connection strength between two nodes when the user continuously gazes at them. It can be adjusted by grid search on the validation set or empirically set (e.g., 0.1 to 0.5) to control the influence of gaze correlation on edge weights. For example, set λ1 to 0.3.

[0101] To establish edge connections between nodes, this paper does not use a fixed distance threshold, but proposes a three-factor function φ that combines spatial position, semantic similarity and gaze behavior continuity. ijThe innovation of the above weight calculation function is to add the attention weight s i , which captures whether the user semantically views two objects as part of a behavioral chain. For example, when observing the coherent behavior of "knife → chopping board → cutting vegetables," even if the objects are spatially separated, as long as the user's gaze behavior is continuous, this formula can enhance the edge connection strength between them, thereby preserving the semantic context of the event.

[0102] In order to reduce the impact of missing information in the area not observed by the user, the present invention obtains potential event nodes according to the audio direction. A potential event node completion mechanism based on audio reflection positions has been introduced:

[0103] Based on audio direction The user's current spatial position p u As the starting point, construct an assumed sound source position r1 is the preset distance constant (such as 5 meters). Generate potential event nodes The fusion semantic feature initialization method is as follows:

[0104]

[0105] represents the set of nodes that are spatially adjacent to the assumed sound source location;

[0106] It is a semantic template of the sound source event constructed from the audio direction (such as the embedding vector of audio tags such as footsteps and shouts);

[0107] μ is the fusion weight (such as μ = 0.7), which is used to control the ratio of structure inference to audio inference. k1 represents The number of nodes in ;

[0108] Based on V t and Constructing spatial semantic graph G t :

[0109]

[0110] is the set of potential event nodes, express The set of introduced edges.

[0111] It integrates three dimensions of information sources: (1) image + gaze-driven entity structure; (2) gaze path-guided semantic connection relationships; and (3) spatial audio-guided event completion nodes. This graph not only represents the relationship between the objects the user is gazing at in the current frame, but also predicts "unseen interaction hotspots" in the scene through an audio inversion mechanism, thereby solving the problem of traditional recognition systems' lack of understanding of invisible events in VR environments. It also provides modeling support for the spatial structure integrity and semantic perception boundary extension of our patent as a whole.

[0112] Step 3: Combine user gaze behavior with audio direction Construct a dynamic semantic graph that integrates semantic content, spatial structure, and perceived events. This step not only elevates the system from "what it sees" to "understanding the structure of the scene it is in," but also enables the system to "recognize unseen but ongoing events" through the unique spatial audio modeling mechanism in the scene. In a virtual reality environment, visual perception has perspective occlusion, while sound has spatial penetration. Traditional graph construction methods often rely on point clouds or geometric relationships, ignoring the user's active perception path and auditory input. This step drives the generation of the graph from the user's perspective movement, visual semantic understanding, and audio direction, improving the integrity, real-time performance, and reasoning capabilities of scene understanding.

[0113] Furthermore, step 4 includes:

[0114] In a graph, a node can be one of three types:

[0115] Entity observation node: The object node extracted from the user's gaze area, which has the fusion semantic features of the image source Spatial position p i , gaze weight s i ;

[0116] Gaze path derived connection nodes: Behavior path connection nodes constructed by gaze direction changes in step 3, which are related to time sequence;

[0117] Potential event nodes By audio direction The event prompt node calculated by spatial projection has a spatial position of Semantics are fused by neighbor inference;

[0118] During the graph propagation process, the edge e ij The information propagation introduces the type weight correction term ψ ij :

[0119]

[0120] in:

[0121] φ ij For edge e ij The weight of

[0122] It is an indicator function for the edge of the gaze behavior chain. For any edge, if the nodes corresponding to its two endpoints are from two consecutive frames and their spatial positions are highly consistent in the user's gaze direction (that is, the angle between the direction of the line connecting the two points and the direction of the user's head is less than 15 degrees), then the edge is considered to be a gaze behavior chain connection and the output value is 1, otherwise it is 0;

[0123] E gaze represents the gaze path;

[0124] Is a judgment function for whether node j is a potential event node. When constructing the graph, the system generates a unique potential event node based on the spatial position extrapolated from the audio direction. Therefore, it is only necessary to determine whether the current node's position is the same as the position extrapolated from the audio direction or the distance is less than 0.5 meters to confirm whether it is a potential event node. If so, the output is 1, otherwise 0;

[0125] λ2 and λ3 are modulation factors for enhancing the influence of gaze path propagation and potential event propagation, respectively; for example, λ2 = 0.2, λ3 = 0.5;

[0126] ψ ij Combined with structural source dynamic regulation;

[0127] Based on ψ ij Perform node updates, including:

[0128] The node update rules are as follows:

[0129]

[0130] in:

[0131] The initial state of the node is given by the fused semantic features;

[0132] W (l) is the weight matrix of the first layer of the graph convolutional neural network;

[0133] σ(·) is a nonlinear activation function;

[0134] After the graph propagation operation, the output h i To fuse structural context, behavioral structure, and audio-aware embedded representations, we ensure the following:

[0135] Gaze path E gaze Receive special attention and improve the structural consistency of nodes on the user behavior path;

[0136] Potential event nodes It is explicitly enhanced to generate potential event diffusion paths with neighboring entity nodes;

[0137] The representations output by all nodes are not labeled but are only representation vectors for further use by subsequent behavior control modules.

[0138] The number of layers of the graph convolutional neural network is L, and the value of L can be 3.

[0139] Through multiple rounds of propagation and aggregation of node states through graph neural networks, the final output is a semantic embedding representation of each node, which is used for subsequent event recognition and system response logic. Unlike traditional GNNs, the graph structure modeled in this step is not based purely on geometric adjacency relationships. Instead, it integrates three patented structural sources: temporal paths driven by gaze behavior, semantic connections guided by image semantic similarity, and complete node connections guided by spatial audio. This improves the semantic consistency, behavioral continuity, and sudden event sensitivity of node state encoding.

[0140] Furthermore, step 5 includes:

[0141] The embedding activation level Act(h i ) to construct candidate regions for indicators The activation score uses the following structure-aware function:

[0142]

[0143] in:

[0144] is the mean of the embedding representations of node i’s neighboring nodes;

[0145] s i is the node’s attention weight, is the average of the attention weights of the neighboring nodes of node i;

[0146] σ(·) is the standard deviation function, which is used to measure the fluctuation of gaze;

[0147] γ(p i ) is a spatial boundary penalty item, which is used to reflect whether the node is close to the edge of the user's field of view. It is calculated by whether the angle between the node position and the user's current position falls outside the current field of view cone. If it exceeds the set viewing angle range (such as horizontal field of view ±60°, vertical field of view ±40°), it is assigned a value of 1, otherwise it is 0, which is used to penalize nodes that fall outside the boundary of the user's field of view.

[0148] λ4 is a parameter used to control the sensitivity of gaze shift, and λ5 is a parameter used to control the penalty of spatial expansion;

[0149] The above-mentioned structure perception function takes into account the combined effects of semantic anomalies, gaze changes and spatial position, and can screen out graph structure hot spots that "although not paid attention to, have high event potential".

[0150] Act(h i ) is higher than the set threshold τ a The nodes are classified into candidate regions

[0151] For each node i in the candidate area, the audio direction driven priority weighting factor ω is further introduced i :

[0152]

[0153] in:

[0154] The audio direction obtained in step 1;

[0155] p i is the spatial position of node i, p u The user's current spatial location;

[0156] cos(·) is the cosine similarity calculation function;

[0157] ρ1 and ρ2 are the proportions of control structure activation and sound priority, respectively;

[0158] This priority weighting factor can effectively improve the feedback level of the area "not seen by the user but audio is perceived" in the system control;

[0159] ω i Sort from large to small and take the top-K nodes as the response node set elements, and for each node Assign response level L i , L i Used to decide what response behavior the system should take. The Top-K value can be 3.

[0160] The classification is constructed based on the following logic function (example):

[0161] If s i <0.3 and It is the “blind spot”, and L i =3;

[0162] If s i >0.7 and Act(h i )>0.8, indicating that the semantics of the user's attention object changes, and L i =2;

[0163] If there are multiple ω in the neighborhood j >τ, then the area is regarded as a “potential event gathering area” and assigned L i =3;

[0164] τ is the priority factor threshold used to filter “high priority nodes”, which is generally set to ω j 90% of the maximum value;

[0165] Otherwise assign L i =1.

[0166] For example:

[0167] L i =3: Immersive guidance behavior (such as automatically and smoothly rotating the view to guide the user to observe the area);

[0168] L i =2: semantic prompting behavior (e.g., voice broadcasting the semantics of the event represented by the node, such as “someone is approaching from behind”);

[0169] L i =1: Mild prompt behavior (such as enhanced light source in the area, flickering edges, etc.);

[0170] The system response module responds to different L i Configuring specific feedback behaviors is driven by standard VR platform interfaces (e.g., controller interfaces, audio prompt interfaces, and HUD component updates in Unity and UE4). It's important to emphasize that this control mechanism doesn't "identify" or "reclassify" the nodes themselves. Instead, it performs multi-factor scoring and decision-making based entirely on the embedded representations obtained in Step 4 and system context signals. This allows for lightweight reasoning, fast response, and interpretable engineering features.

[0171] Traditional VR systems often have pre-set interaction processes and cannot quickly adjust to changes in environmental conditions. However, our system can proactively prompt or intervene even when the user is not actively observing, through semantic embedding perception and event state modeling. To this end, step 5 combines the output of step 4 with the gaze weights, spatial positions, and audio directions of steps 1 to 3 to generate a set of response strategies. And the specific behavioral feedback is determined through the hierarchical control module.

[0172] Furthermore, step 6 includes:

[0173] Introducing the user in the feedback time window [t+1,t+τ f ] behavioral trajectory data within Used to determine the degree of match between system response and user behavior;

[0174] Calculate the consistency judgment value of each node in the response node set separately, including:

[0175] Define a structure-feedback consistency discriminant function to calculate the consistency discriminant value η(i) of node i:

[0176]

[0177] h i is the embedding representation of node i, is the mean of the embedding representation of the neighboring nodes of node i, cos(·) is the cosine similarity calculation function, p i is the spatial position of node i, represents the spatial position of the user at time point t+k, t is the time point when the system executes the response behavior, σ is the distance attenuation control coefficient, which is used to determine the degree of influence of spatial error on the score (e.g. σ = 1.5 meters), τ f is the length of the feedback time window (e.g., set to 1 second), κ1 and κ2 are the weight coefficients of the structural consistency term and the behavior following term, respectively (e.g., κ1 = 0.6, κ2 = 0.4); in step 6,

[0178] η(i)∈[0,1]: The lower the value, the more inconsistent the response node is in terms of structure and behavior feedback, and the more it needs to be updated.

[0179] Update the node embedding representation based on the consistency discriminant value, including:

[0180] All η(i)<τ i Node, τ i For the set discriminant function threshold (e.g. 0.5), a local representation correction term is constructed to bring its embedded representation closer to the neighborhood mean and retain the user feedback influence factor δ i :

[0181]

[0182] is the updated embedding representation of node i; is the maximum update amplitude (e.g. 0.1), which ensures that the worse the feedback, the stronger the update.

[0183] The update does not change the parameters of the entire model, but only updates the node embedding representation in the graph embedding cache, which will have an impact in the next inference;

[0184] If a node enters the low consistency interval multiple times, it triggers fine-tuning of the structural graph construction parameters (such as edge connection threshold) or the parameters within the GNN layer (such as local weights).

[0185] Furthermore, to prevent unnecessary perturbations, a structural stability controller can be introduced into the system design, which only allows updates to be performed when the local subgraph meets the following conditions:

[0186] The size of the sub-image should not exceed 10% of the original image;

[0187] The difference in subgraph node activation is 1σ above the mean;

[0188] The subgraph center contains at least one potential event node Nodes that are directly connected by edges.

[0189] These conditions are detected by a predictive module, ensuring that structural updates are centralized, stable, and without impacting global connectivity. Ultimately, this mechanism, a key component of the patented technology solution that achieves "long-term learning, self-regulation, and structural awareness," enables the system to not only recognize and respond, but also continuously improve its intelligence as user habits evolve.

[0190] This step enables the model to self-adjust and optimize its internal representation structure based on actual user behavior without relying on external manual annotation. The previous five steps have already formed a complete closed loop of "perception-modeling-inference-response," and this step models the "self-evolution" mechanism of this closed loop. The table step tracks the actual user feedback trajectory after system behavior. While ensuring that the graph structure and the original reasoning path are not destroyed, it performs structure-guided updates to the local node embedding representation, thereby forming an embeddable and continuously running model adaptation path.

[0191] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An AI-based VR scene intelligent recognition method, characterized in that: include: Step 1: When a user uses a VR headset, obtain the image frame I, head posture H, and spatial audio A displayed by the VR headset, process I, H, and A to obtain a sequence of time-aligned triplets, where the time-aligned triplets include the image frame, head posture, and audio direction; Step 2: Obtain the image features of each image frame respectively, and obtain the fused semantic features of each time point based on the image features; Step 3: Construct a spatial semantic map based on the fusion semantic features and audio direction; Step 4: Perform graph propagation operations on the nodes in the spatial semantic graph to obtain the embedded representation of each node in the spatial semantic graph; Step 5: Obtain the response node set based on the embedded representation; Step 6: Update the embedding representation of the nodes in the spatial semantic graph based on the response node set.

2. The AI-based VR scene intelligent recognition method according to claim 1, characterized in that: The head posture H is measured by the IMU of the VR headset; The image frame I is obtained from the video channel of the VR headset.

3. The AI-based VR scene intelligent recognition method according to claim 1, characterized in that: I, H, and A are processed to obtain a time-aligned triple sequence, including: The image frame obtained at time point t is represented as I t , for the head posture H obtained at time point t' t′ Perform linear interpolation resampling to calculate the head posture H at time point t t ; Perform weighted averaging of the direction vectors of all spatial audio within the time window centered at time point t and with a window half-width of δ to obtain the global audio direction Set the global audio direction Project to the local coordinate system to obtain the final normalized audio direction The time-aligned triplet at time point t is The triple sequence is represented as T represents the length of the time series of continuous processing, and t0 represents the starting time point of the time series of continuous processing.

4. The AI-based VR scene intelligent recognition method according to claim 3, characterized in that: Step 2 includes: From the image frame I t Extract the visual feature vector F t ; Set the head posture H t Converted to a three-dimensional gaze direction unit vector hc t ; The fused semantic features of time point p are expressed as 5. The AI-based VR scene intelligent recognition method according to claim 4, characterized in that: Step 3 includes: Combined I t The depth information carried by the object being watched in the image frame is back-projected into three-dimensional space to obtain the spatial position of the object being watched The fusion semantic features at time point t and p t As the node construction element, define the initial node set V of the spatial semantic graph t ; Get potential event nodes based on audio direction Based on V t and Constructing spatial semantic graph G t .

6. The AI-based VR scene intelligent recognition method according to claim 5, characterized in that: The edge set E of the spatial semantic graph t The calculation process of the edge weight in includes: The weight of the edge is calculated based on the spatial position, semantic features and gaze weight of the two nodes corresponding to the edge.

7. The AI-based VR scene intelligent recognition method according to claim 5, characterized in that: Get potential event nodes based on audio direction include: Based on audio direction The user's current spatial position p u As the starting point, construct an assumed sound source position r is the preset distance constant, Generate potential event nodes 8. The AI-based VR scene intelligent recognition method according to claim 6, characterized in that: Step 4 includes: During the graph propagation process, the edge e ij The information propagation introduces the type weight correction term ψ ij : Based on ψ ij Perform node updates.

9. The AI-based VR scene intelligent recognition method according to claim 8, characterized in that: Step 5 includes: The embedding activation level Act(h i ) to construct candidate regions for indicators Act(h i ) is higher than the set threshold τ a The nodes are classified into candidate regions For each node i in the candidate area, the audio direction driven priority weighting factor ω is further introduced i ; ω i Sort from large to small and take the top-K nodes as the response node set elements, and for each node Assign response level L i , L i Used to decide what response behavior the system should take.

10. The AI-based VR scene intelligent recognition method according to claim 9, characterized in that: Step 6 includes: Calculate the consistency discrimination value of each node in the response node set respectively; The embedding representation of the node is updated based on the consistency discriminant value.

Images