A method and system for scene analysis based on virtual reality

Abstract

The present invention discloses a virtual reality-based scene analysis method and system, which relates to the field of virtual reality analysis technology. The method comprises the following steps: using sensor equipment to acquire three-dimensional point cloud data of a scene in real time and preprocessing it; performing multi-scale layered sampling on the three-dimensional point cloud data, applying a masking operation to each layer to obtain visible points and masked points, performing multi-scale neighborhood modeling based on the visible points to extract neighborhood feature vectors, reconstructing masked points through an autoencoder, extracting masked point features, and performing multi-level Transformer feature fusion to obtain multi-scale features; assigning point cloud labels to construct semantic contextual relationships, outputting point features through multi-channel contextual attention weight distribution, and obtaining global features through multi-layer semantic fusion; and performing scene analysis and outputting scene categories by combining multi-scale features and global features through a convolutional neural network. The present invention effectively improves the expressive adequacy and intelligent analysis accuracy of point cloud scenes, and improves the accuracy of scene classification.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality analysis technology, and in particular to a scene analysis method and system based on virtual reality. Background Art

[0002] With the rapid development of virtual reality (VR) technology, scene analysis is playing an increasingly important role in multiple fields, including 3D perception, digital twins, and intelligent manufacturing. As the primary data carrier for scene spatial information, 3D point clouds, thanks to the maturity of multi-sensor technologies such as LiDAR, structured light, and depth cameras, are widely used for realistic reconstruction and intelligent analysis of VR scenes. In recent years, point cloud-based scene analysis methods have evolved from traditional geometric feature extraction and rule-based approaches to the integration of various techniques such as deep learning and graph neural networks. Current mainstream research focuses on tasks such as point cloud segmentation, object detection, scene classification, and relationship parsing. Some methods address the sparsity and disorder of point clouds by introducing strategies such as multi-scale sampling, neighborhood feature aggregation, attention mechanisms, and global context embedding. For example, network architectures such as the PointNet series, DGCNN, and PointTransformer have continuously improved the accuracy and efficiency of point cloud structure understanding. However, current technologies still have shortcomings. Existing technologies struggle to fully represent the complex spatial relationships of scenes when analyzing point clouds. They also lack hierarchical, multi-scale, and dynamically adaptive processing mechanisms, limiting their ability to represent multi-category semantic structures. Summary of the Invention

[0003] In view of the above existing problems, the present invention is proposed.

[0004] Therefore, the present invention provides a scene analysis method and system based on virtual reality to solve the problems that the existing technology is difficult to fully express the complex spatial relationships of the scene when analyzing point clouds, lacks hierarchical, multi-scale and dynamic adaptive processing mechanisms, and has limited ability to express multi-category semantic structures.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a scene analysis method based on virtual reality, which comprises:

[0007] Use sensor equipment to obtain scene 3D point cloud data in real time and perform pre-processing;

[0008] Multi-scale layered sampling is performed on 3D point cloud data. A neighborhood radius is set for each layer of sampling points, and a masking operation is applied to each layer to obtain visible points and masked points. The neighborhood scale is calculated based on the neighborhood radius to generate a neighborhood point cloud set. Multi-scale neighborhood modeling is performed based on the visible points to extract neighborhood feature vectors. The neighborhood feature vectors are encoded using a convolutional autoencoder and then fused to obtain the final embedded features. The masked points are reconstructed using the autoencoder combined with the final embedded features of the visible points. The reconstructed embedded features of the masked points are extracted and then fused with multi-level Transformer features to obtain multi-scale features.

[0009] The point cloud is assigned point labels to build semantic contextual relationships. The contextual enhancement features of the output points are weighted and aggregated through multi-channel contextual attention weights. The fused high-order features of the points are obtained through multi-layer semantic splicing. The global features are obtained by global pooling operation after fusion through a two-layer perceptron and fusion channel weights.

[0010] The convolutional neural network is used to combine multi-scale features and global features to perform scene analysis, output scene categories, and visualize the analysis results.

[0011] As a preferred solution of the scene analysis method based on virtual reality of the present invention, wherein: the mask point reconstruction by combining the final embedding features of the visible points with the autoencoder refers to obtaining the final embedding feature set of each layer visible Record, search for the nearest visible point for each mask point in each layer and extract the final embedding features of the visible point , the visible points are finally embedded in the features and the final set of embedded features below layer i Input decoder, and fit the mask point space coordinates through the decoder And Chamfer Distance is used as the loss metric.

[0012] As a preferred solution of the virtual reality-based scene analysis method described in the present invention, wherein: the point cloud point labels are assigned to construct semantic contextual relationships, and the context-enhanced features of the output points are weightedly aggregated through multi-channel contextual attention weights. K-means clustering is performed on the pre-processed three-dimensional point cloud data, and cluster labels are assigned to each point according to the clustering results. , embed the cluster label into the aggregated features of each point, and calculate the multi-channel context attention weight of each point ;

[0013] The context-enhanced features of each point are obtained by weighted aggregation through multi-channel context attention weights .

[0014] As a preferred solution of the scene analysis method based on virtual reality described in the present invention, wherein: the fusion high-order features of the points are obtained by splicing through multi-layer semantics, and the global features are obtained by global pooling operation after fusion through a two-layer perceptron and fusion channel weights. It means that the context-enhanced features of each point in each layer are obtained based on the sampling point set of each layer, and the context-enhanced features of the corresponding points of all layers are spliced according to the corresponding relationship of the sampling points to obtain each point. Fusion of high-level features , calculate the channel mean z of the point cloud data;

[0015] Generate fusion channel weight s through a two-layer perceptron;

[0016] The fusion channel weight s is used to fuse the high-order features of each point Perform weighted output ;

[0017] Weighted output of all point cloud points The maximum pooling and average pooling operations are respectively used to obtain the global scene feature vector. .

[0018] As a preferred embodiment of the virtual reality-based scene analysis method of the present invention, the method of performing scene analysis and outputting scene categories by combining multi-scale features and global features with a convolutional neural network refers to constructing a convolutional neural network, setting the input as the multi-scale features of point cloud data and the global scene feature vector, and outputting the scene category;

[0019] Train the convolutional neural network using the training data set, define the loss function and use the Adam optimizer to iteratively optimize the convolutional neural network parameters until the loss converges;

[0020] The multi-scale features and global scene feature vectors are input into the trained convolutional neural network to obtain the scene category.

[0021] As a preferred solution of the virtual reality-based scene analysis method described in the present invention, the use of sensor equipment to obtain the three-dimensional point cloud data of the scene in real time and perform preprocessing refers to using a binocular stereo camera and an inertial measurement unit to obtain the three-dimensional point cloud data of the scene in real time and perform denoising and normalization processing.

[0022] As a preferred solution of the virtual reality-based scene analysis method described in the present invention, the visual display of the analysis results refers to forming a visual table to display the scene categories obtained by analysis and the collected point cloud data, and synchronously storing the analysis results and point cloud data in a database for backup.

[0023] In a second aspect, the present invention provides a scene analysis system based on virtual reality, comprising:

[0024] Point cloud acquisition module, used to collect scene point cloud data in real time through sensors and perform preprocessing;

[0025] The multi-scale feature extraction module is used to perform multi-scale layered sampling on 3D point cloud data and apply a masking operation to each layer to obtain visible points and masked points. Multi-scale neighborhood modeling is performed based on the visible points to extract neighborhood feature vectors. Masked points are reconstructed through an autoencoder, and the extracted masked point features are fused with multi-level Transformer features to obtain multi-scale features.

[0026] The global feature extraction module is used to assign point cloud point labels to construct semantic contextual relationships, output point features through multi-channel contextual attention weight distribution, and obtain global features through multi-layer semantic fusion;

[0027] The scene analysis module is used to perform scene analysis by combining multi-scale features and global features through convolutional neural networks, output scene categories, and visualize the analysis results.

[0028] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the virtual reality-based scene analysis method as described in the first aspect of the present invention is implemented.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the virtual reality-based scene analysis method as described in the first aspect of the present invention is implemented.

[0030] The beneficial effects of the present invention are as follows: the present invention samples scene point clouds in real time, adopts multi-scale layered sampling and masking mechanism, autoencoder reconstruction, hierarchical Transformer multi-scale feature fusion and multi-channel context attention-driven global feature generation, thereby effectively improving the expressiveness and intelligent analysis accuracy of point cloud scenes and improving the accuracy of scene classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is a flowchart of the virtual reality-based scene analysis method in Example 1.

[0033] Figure 2 This is a structural diagram of the virtual reality-based scene analysis system in Example 1. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0036] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0037] Example 1, with reference to Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a scene analysis method based on virtual reality, comprising the following steps:

[0038] S1, using sensor equipment to obtain scene 3D point cloud data in real time and perform preprocessing;

[0039] Specifically, using sensor equipment to obtain three-dimensional point cloud data of the scene in real time and performing preprocessing means using a binocular stereo camera and an inertial measurement unit to obtain three-dimensional point cloud data of the scene in real time and perform denoising and normalization processing.

[0040] S2. Perform multi-scale layered sampling on the three-dimensional point cloud data. Set a neighborhood radius for each layer of sampling point set, apply a masking operation to each layer to obtain visible points and masked points, calculate the neighborhood scale based on the neighborhood radius to generate a neighborhood point cloud set, perform multi-scale neighborhood modeling based on the visible points to extract neighborhood feature vectors, and use a convolutional autoencoder to encode the neighborhood feature vectors and fuse them to obtain the final embedded features. Reconstruct the masked points through the autoencoder combined with the final embedded features of the visible points, extract the reconstructed embedded features of the masked points, and perform multi-level Transformer feature fusion to obtain multi-scale features.

[0041] Specifically, multi-scale layered sampling is performed on the three-dimensional point cloud data, a neighborhood radius is set for each layer of sampling point set, and a mask operation is applied to each layer to obtain visible points and mask points. The neighborhood scale is calculated in combination with the neighborhood radius to generate a neighborhood point cloud set. Multi-scale neighborhood modeling is performed based on the visible points to extract the neighborhood feature vector, and the neighborhood feature vector is encoded using a convolutional autoencoder and then fused to obtain the final embedded feature. The mask point is reconstructed by combining the final embedded feature of the visible point through the autoencoder, and the reconstructed embedded feature of the mask point is extracted to perform multi-level Transformer feature fusion to obtain multi-scale features. According to the preprocessed three-dimensional point cloud data, the sampling point set of each layer is obtained by Farthest Point Sampling sampling. :

[0042]

[0043] in is the previous layer point set, initially the pre-processed 3D point cloud dataset, is the number of target sampling points in the i-th layer, To iterate over the input point set The most distant representative points;

[0044] Set the neighborhood radius of each layer for the sampling point set of each layer :

[0045]

[0046] in is the initial neighborhood radius, is the radius increasing coefficient;

[0047] In each layer of sampling point set, random mask operation is performed according to the set mask ratio to obtain the visible point subset and the subset of mask points :

[0048]

[0049] in is the mask ratio of the i-th layer, is a random mask function;

[0050] The mask ratio is set according to the formula, which is expressed as:

[0051]

[0052] in is the lowest layer mask ratio, The increment for each layer;

[0053] According to the neighborhood radius of each layer Combine multiple scale coefficients to generate the neighborhood scale of each layer :

[0054]

[0055] in is a multi-scale coefficient, such as 1, 2, 4, etc., and m is the scale number;

[0056] The visible point subset of each layer is calculated by neighborhood scale Every visible point Generate neighborhood point cloud collection :

[0057]

[0058] in is the jth point in the sampling point set of the i-th layer;

[0059] According to each point in the neighborhood point cloud set and the visible points The relative coordinates of each scale are used to construct the neighborhood feature vector :

[0060]

[0061] in is the relative coordinate between the point pairs;

[0062] The convolutional autoencoder is used to encode the neighborhood feature vector of each visible point at each scale to obtain deep embedding features. , weighted fusion is used to fuse the deep embedding features of each scale to obtain the final embedding features of each visible point ;

[0063] Get the final set of embedding features visible at each layer Record, search for the nearest visible point for each mask point in each layer and extract the final embedding features of the visible point , the visible points are finally embedded in the features and the final set of embedded features below layer i Input decoder, and fit the mask point space coordinates through the decoder And adopt ChamferDistance as the loss metric:

[0064]

[0065] in is the set of mask points obtained by fitting, and b are the fitted mask points and the true mask points respectively, for loss;

[0066] Obtain the reconstruction embedding features based on the reconstruction mask points obtained by fitting And determine the neighborhood and calculate the neighborhood localization attention score of each point in each layer :

[0067]

[0068] in and are query vector and key vector respectively, is the key vector of the p-th point, is the projection vector dimension, determined by the embedded features;

[0069] Use the neighborhood localization attention score of each point to perform weighted aggregation on each point to obtain the fusion feature output of the point :

[0070]

[0071] in is a value vector;

[0072] Calculate the Squeeze-Excitation weight of the fusion feature output of each layer point And output the fusion feature Perform weighted output to obtain aggregated features :

[0073]

[0074]

[0075] in is channel-by-channel multiplication;

[0076] Concatenate the aggregate features of all points in each layer to obtain multi-scale features .

[0077] Multi-scale layered sampling and masking operations not only form a hierarchical spatial perception network, but also provide highly robust input for irregular point cloud data in real application scenarios. For example, common occlusion, missing or density change problems in virtual reality scenes can be converted into effective information redundancy and completion relationships under the mechanism of the present invention, leaving redundant space for multi-level feature expression. At the same time, by setting random masks rather than fixed-point removal, sampling bias can be prevented to the greatest extent, making subsequent feature learning more generalizable. Secondly, the combination of multi-scale neighborhood modeling and relative coordinate feature extraction lays the foundation for the efficient representation of local and global geometric relationships of point clouds, extracting neighborhood structures at different scales, not only capturing the microscopic details of point clouds, but also integrating macroscopic structures to achieve layered information coupling, which has inherent advantages in applications such as complex scene restoration and virtual object recognition.

[0078] By coordinating the mask mechanism and the autoencoder network, feature restoration and spatial completion are achieved in cases of undersampling, occlusion, and data missing. This mechanism not only improves the accuracy of virtual scene restoration, but also greatly improves the discriminative ability of feature expression through deep learning of embedded features, providing a solid foundation for subsequent high-level tasks (such as scene semantic segmentation, object detection, etc.). The nearest visible point feature and multi-layer embedding set input decoder take into account local consistency and global contextual connections, avoiding the shortcomings of traditional interpolation that are easily affected by outliers. In addition, based on the features of the reconstructed mask points, a localized attention mechanism is introduced, which can accurately focus on areas with close feature connections in the point cloud, suppress interference and improve discrimination ability, and significantly make up for the shortcomings of point cloud analysis in the dynamic fusion of global and local information.

[0079] The Squeeze-Excitation weighting mechanism and multi-scale feature splicing provide a multi-level, full-channel adaptive information aggregation method for the final scene understanding task. This step not only enhances the expression of information at each scale and semantic channel, but also adaptively allocates feature weights, improving the model's generalization and applicability for different task objectives.

[0080] S3: Assign point labels to point clouds to build semantic contextual relationships, perform weighted aggregation through multi-channel contextual attention weights to output contextual enhancement features of the output points, and obtain fused high-order features of the points through multi-layer semantic splicing. After fusing through a two-layer perceptron and fusion channel weights, a global pooling operation is used to obtain global features.

[0081] Specifically, point cloud point labels are assigned to build semantic contextual relationships, and the contextual enhancement features of the output points are weighted and aggregated through multi-channel contextual attention weights. K-means clustering is performed on the preprocessed 3D point cloud data, and cluster labels are assigned to each point based on the clustering results. , embed the cluster label into the aggregated features of each point, and calculate the multi-channel context attention weight of each point :

[0082]

[0083] in is the projection weight, MLP is the multi-layer perceptron, is the aggregated feature after embedding the cluster label of the i-th point, and are the aggregate features after embedding cluster labels of the jth and kth points in the neighborhood of the i-th point respectively;

[0084] The context-enhanced features of each point are obtained by weighted aggregation through multi-channel context attention weights :

[0085]

[0086] in To improve the feature expression ability of neighborhood points after transformation through MLP.

[0087] K-means clustering and label embedding break the traditional point cloud processing model that only relies on Euclidean space distribution or single attribute aggregation. By semantically dividing the point cloud space and embedding high-dimensional labels, each point can obtain not only its own attributes and position when updating its features, but also perceive its distribution in a higher-dimensional semantic space. This processing is particularly critical for complex situations where there are category transitions, blurred boundaries, or multiple targets nested within the scene. For example, in virtual reality scene analysis, adjacent objects with obvious morphological differences can be distinguished through semantic labels to reduce cross-category misclustering. For point clouds in multi-scale, dense or sparse areas, semantic labels provide a more stable intermediary for the attribution and relationship network of points, effectively improving the robustness of subsequent context perception and feature transfer. This concept is significantly better than traditional local methods such as Jaccard or distance neighborhood alone.

[0088] By learning the point-to-point "correlation weight" in the neighborhood with cluster labels as the dominant feature, each point is ultimately expressed as an information expression body that integrates its own strongly correlated points. Unlike the current mainstream spatial attention or channel attention modeling methods, the multi-channel solution can strengthen information interaction in parallel in both feature channel and neighborhood semantic spaces. Artificially designed aggregation weight patterns often have performance bottlenecks due to the lack of global context and difficulty in dynamically adapting to scene changes. The solution of the present invention can achieve adaptive weighting for semantic differences and spatial structures, fully highlighting the point features with key contributions in the context environment, automatically suppressing redundant or low-correlation information, and significantly reducing irrelevant noise interference.

[0089] Furthermore, the fusion high-order features of the points are obtained by splicing multiple layers of semantics, and the global pooling operation is used to obtain the global features after fusing through the double-layer perceptron and the fusion channel weights. The context-enhanced features of each point in each layer are obtained based on the sampling point set of each layer, and the context-enhanced features of the corresponding points of all layers are spliced according to the corresponding relationship of the sampling points to obtain the global features of each point. Fusion of high-level features , calculate the channel mean z of the point cloud data:

[0090]

[0091] Where N is the number of point cloud points;

[0092] Generate fusion channel weight s through a two-layer perceptron:

[0093]

[0094] in and are the weights of the fully connected layer, and is the bias, is the ReLU activation function, is the Sigmoid function;

[0095] The fusion channel weight s is used to fuse the high-order features of each point Perform weighted output :

[0096]

[0097] in is channel-by-channel multiplication;

[0098] Weighted output of all point cloud points The maximum pooling and average pooling operations are respectively used to obtain the global scene feature vector. .

[0099] By obtaining context-enhanced features on each layer of sampling point sets, it is possible to be compatible with semantic and structural information at different spatial scales, effectively covering the two types of difficulties that may exist in point clouds: local high complexity and loose global connections. Compared with traditional methods that only use a single scale, a single semantic or basic geometric features, the present invention grafts multi-level expressions onto a single point, significantly improving the semantic integrity and discrimination of point cloud analysis. For example, in large-scale indoor and outdoor scenes, semantic clues from complex object components, overall structures to local details are all captured together, greatly reducing information gaps and spatial ambiguity, and having irreplaceable comprehensiveness in practical applications such as multi-target discrimination and anomaly detection.

[0100] By automatically generating global channel weights through a two-layer perceptron, the importance of feature channel representation is adaptively controlled. This mechanism effectively highlights feature channels that contribute significantly to discriminability while suppressing redundant or noisy features, achieving an optimal balance between semantic compression and multidimensionality preservation. The benefit is that the model automatically extracts optimal information based on the actual data distribution in various scene segmentation, object recognition, and high-level spatial reasoning tasks, achieving higher generalization and robustness.

[0101] S4. Use convolutional neural networks to combine multi-scale features and global features to perform scene analysis, output scene categories, and visualize the analysis results.

[0102] Specifically, performing scene analysis and outputting scene categories by combining multi-scale features and global features through a convolutional neural network refers to constructing a convolutional neural network, setting the input as the multi-scale features of point cloud data and the global scene feature vector, and outputting the scene category;

[0103] Train the convolutional neural network using the training data set, define the loss function and use the Adam optimizer to iteratively optimize the convolutional neural network parameters until the loss converges;

[0104] The multi-scale features and global scene feature vectors are input into the trained convolutional neural network to obtain the scene category.

[0105] Convolutional neural networks, as a structured deep learning method, differ from the point-by-point or neighborhood-by-neighborhood static computations of traditional point cloud analysis. Their multi-core convolutional layers automatically explore high-order local and global relationships in spatial relationships, making them particularly well-suited for feature extraction and discrimination in highly redundant, disordered, and topologically unstructured 3D point cloud data. By integrating multi-scale features with global scene feature vectors into the network, the model can balance detailed distinctions with global structural understanding within the same inference process, significantly improving recognition accuracy and generalization capabilities for complex scene categories.

[0106] Using a repeatedly optimized convolutional neural network as a scene classifier essentially achieves end-to-end high-level semantic mapping. This outputs world-class classification of complex environments by integrating multi-scale input with global features. This is particularly important for fields such as virtual reality, digital twins, and smart city perception.

[0107] Furthermore, visually displaying the analysis results means displaying the scene categories obtained through analysis and the collected point cloud data in a visual table, and simultaneously storing the analysis results and point cloud data in a database for backup.

[0108] This embodiment also provides a virtual reality-based scene analysis system, including:

[0109] Point cloud acquisition module, used to collect scene point cloud data in real time through sensors and perform preprocessing;

[0110] The multi-scale feature extraction module is used to perform multi-scale layered sampling on 3D point cloud data and apply a masking operation to each layer to obtain visible points and masked points. Multi-scale neighborhood modeling is performed based on the visible points to extract neighborhood feature vectors. Masked points are reconstructed through an autoencoder, and the extracted masked point features are fused with multi-level Transformer features to obtain multi-scale features.

[0111] The global feature extraction module is used to assign point cloud point labels to construct semantic contextual relationships, output point features through multi-channel contextual attention weight distribution, and obtain global features through multi-layer semantic fusion;

[0112] The scene analysis module is used to perform scene analysis by combining multi-scale features and global features through convolutional neural networks, output scene categories, and visualize the analysis results.

[0113] This embodiment also provides a computer device suitable for the scenario of a virtual reality-based scene analysis method, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the virtual reality-based scene analysis method proposed in the above embodiment.

[0114] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0115] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the virtual reality-based scene analysis method proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0116] The present invention samples scene point clouds in real time, adopts multi-scale layered sampling and masking mechanism, autoencoder reconstruction, hierarchical Transformer multi-scale feature fusion and multi-channel context attention-driven global feature generation, effectively improving the expressiveness of point cloud scenes and the accuracy of intelligent analysis, and improving the accuracy of scene classification.

[0117] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A scene analysis method based on virtual reality, characterized by: include, Use sensor equipment to obtain scene 3D point cloud data in real time and perform pre-processing; Multi-scale layered sampling is performed on 3D point cloud data. A neighborhood radius is set for each layer of sampling points, and a masking operation is applied to each layer to obtain visible points and masked points. The neighborhood scale is calculated based on the neighborhood radius to generate a neighborhood point cloud set. Multi-scale neighborhood modeling is performed based on the visible points to extract neighborhood feature vectors. The neighborhood feature vectors are encoded using a convolutional autoencoder and then fused to obtain the final embedded features. The masked points are reconstructed using the autoencoder combined with the final embedded features of the visible points. The reconstructed embedded features of the masked points are extracted and then fused with multi-level Transformer features to obtain multi-scale features. The point cloud is assigned point labels to build semantic contextual relationships. The contextual enhancement features of the output points are weighted and aggregated through multi-channel contextual attention weights. The fused high-order features of the points are obtained through multi-layer semantic splicing. The global features are obtained by global pooling operation after fusion through a two-layer perceptron and fusion channel weights. The convolutional neural network is used to combine multi-scale features and global features to perform scene analysis, output scene categories, and visualize the analysis results.

2. The virtual reality-based scene analysis method according to claim 1, wherein: The mask point reconstruction by combining the final embedding features of the visible points with the autoencoder refers to obtaining the final embedding feature set visible at each layer. Record, search for the nearest visible point for each mask point in each layer and extract the final embedding features of the visible point , the visible points are finally embedded in the features and the final set of embedded features below layer i Input decoder, and fit the mask point space coordinates through the decoder And Chamfer Distance is used as the loss metric.

3. The virtual reality-based scene analysis method according to claim 2, wherein: The point cloud point labels are assigned to construct semantic contextual relationships, and the contextual enhancement features of the output points are weighted and aggregated through multi-channel contextual attention weights. K-means clustering is performed on the pre-processed three-dimensional point cloud data, and cluster labels are assigned to each point according to the clustering results. , embed the cluster label into the aggregated features of each point, and calculate the multi-channel context attention weight of each point ; The context-enhanced features of each point are obtained by weighted aggregation through multi-channel context attention weights .

4. The virtual reality-based scene analysis method according to claim 3, wherein: The fusion high-order features of the points are obtained by splicing multiple layers of semantics, and the global features are obtained by global pooling operation after fusion through the two-layer perceptron and fusion channel weights. It refers to obtaining the contextual enhancement features of each point in each layer based on the sampling point set of each layer, and splicing the contextual enhancement features of the corresponding points of all layers according to the corresponding relationship of the sampling points to obtain each point. Fusion of high-level features , calculate the channel mean z of the point cloud data; Generate fusion channel weight s through a two-layer perceptron; The fusion channel weight s is used to fuse the high-order features of each point Perform weighted output ; Weighted output of all point cloud points The maximum pooling and average pooling operations are respectively used to obtain the global scene feature vector. .

5. The virtual reality-based scene analysis method according to claim 4, wherein: The outputting of the scene category by combining the multi-scale features and the global features with the convolutional neural network for scene analysis refers to constructing a convolutional neural network, setting the input as the multi-scale features of the point cloud data and the global scene feature vector, and outputting the scene category; Train the convolutional neural network using the training data set, define the loss function and use the Adam optimizer to iteratively optimize the convolutional neural network parameters until the loss converges; The multi-scale features and global scene feature vectors are input into the trained convolutional neural network to obtain the scene category.

6. The virtual reality-based scene analysis method according to claim 5, wherein: The use of sensor equipment to obtain three-dimensional point cloud data of the scene in real time and perform preprocessing refers to using a binocular stereo camera and an inertial measurement unit to obtain three-dimensional point cloud data of the scene in real time and perform denoising and normalization processing.

7. The virtual reality-based scene analysis method according to claim 6, wherein: The visual display of the analysis results refers to forming a visual table of the scene categories obtained by analysis and the collected point cloud data for display, and synchronously storing the analysis results and the point cloud data in a database for backup.

8. A virtual reality-based scene analysis system, based on the virtual reality-based scene analysis method according to any one of claims 1 to 7, characterized in that: include, Point cloud acquisition module, used to collect scene point cloud data in real time through sensors and perform preprocessing; The multi-scale feature extraction module is used to perform multi-scale layered sampling on 3D point cloud data and apply a masking operation to each layer to obtain visible points and masked points. Multi-scale neighborhood modeling is performed based on the visible points to extract neighborhood feature vectors. Masked points are reconstructed through an autoencoder, and the extracted masked point features are fused with multi-level Transformer features to obtain multi-scale features. The global feature extraction module is used to assign point cloud point labels to construct semantic contextual relationships, output point features through multi-channel contextual attention weight distribution, and obtain global features through multi-layer semantic fusion; The scene analysis module is used to perform scene analysis by combining multi-scale features and global features through convolutional neural networks, output scene categories, and visualize the analysis results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the virtual reality-based scene analysis method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the virtual reality-based scene analysis method according to any one of claims 1 to 7 are implemented.

Images