Virtual reality space-time data compression and semantic enhancement method based on millimeter wave point cloud

By adopting a two-stage semantic recognition method based on millimeter wave point cloud in virtual reality, the problem of large amount of point cloud data processing and high memory demand in virtual reality is solved, efficient semantic recognition and data processing are achieved, and virtual reality is optimized Performance and resource utilization of semantic recognition.

CN119002702AActive Publication Date: 2024-11-22SOUTHEAST UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411107384.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-22
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

The processing of millimeter-wave point cloud data in virtual reality has problems such as large amount of computing, high memory requirements and insufficient robustness, especially in real-time semantic recognition and data transmission and storage.

Method used

Using virtual reality spatiotemporal data compression and semantic enhancement methods based on millimeter wave point cloud, including positioning stage and focus stage, each point is assigned attention score through point cloud transformer, group adjacent points, calculate comprehensive scores, and use neighbors. The domain global semantic attention mechanism is used for early recognition, and the attention mechanism is applied in the focus stage to remove semantic-independent points, optimize computing resources and accuracy.

Benefits of technology

The accuracy and computing resources optimization in virtual reality semantic recognition are realized, which reduces computing costs and memory resource consumption, improves data processing efficiency and feature reuse, and significantly improves the performance of virtual reality semantic recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002702A_ABST
    Figure CN119002702A_ABST
Patent Text Reader

Abstract

The invention discloses a virtual reality space-time data compression and semantic enhancement method based on millimeter wave point clouds, which at least comprises a positioning stage and a focusing stage, in the positioning stage, a point cloud converter distributes an attention score for each point, groups adjacent points in the point clouds, calculates a comprehensive score of the whole group, and when the semantic resolution is not clear, the focusing stage is started; a neighborhood global semantic attention (NGSA) mechanism searches a region with a high semantic score, early recognition is performed by analyzing original point cloud data, and features are reused by using an intelligent strategy; in the focusing stage, an attention mechanism is applied to the key point cloud area selected in the positioning stage for focusing recognition, the importance degree of points in the area is determined, and the points irrelevant to semantics are removed. According to the method, optimization of accuracy and computing resources is realized, and efficient data processing and feature reuse are realized by utilizing a point cloud converter architecture.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual reality and artificial intelligence, and in particular relates to a virtual reality spatiotemporal data compression and semantic enhancement method based on millimeter wave point cloud. Background Art

[0002] Semantic recognition in virtual reality plays an important role in many application areas, such as entertainment, shopping, healthcare, and education. The traditional method is to extract semantic information by identifying and tracking the user's movements through the inertial measurement unit and camera in the virtual reality headset and controller. However, this traditional method often requires a lot of computing resources and has significant defects in data transmission and storage.

[0003] In recent years, with the rapid development of sensor technology, especially the progress of millimeter wave radar technology, new possibilities have been brought to the field of virtual reality. Specifically, millimeter wave radar can not only penetrate certain obstacles and capture information of obscured objects, but also has the characteristics of high sensitivity and low power consumption. However, the high dimensionality and sparsity of point cloud data make the direct processing and analysis of these data extremely complex and computationally intensive. At present, the methods based on visual transformers and point cloud transformers have problems such as large computational complexity, high memory requirements, and insufficient robustness when processing millimeter wave point cloud data.

[0004] Emerging integrated millimeter-wave radars are low-cost and small (about 8 cm), so they can be easily integrated into virtual reality environments for semantic recognition tasks. In addition, recent applications have demonstrated the potential of radar in providing various human perception capabilities. Millimeter-wave radars have the advantages of strong penetration, high environmental adaptability, and privacy protection. Compared with traditional optical sensors, millimeter-wave radars can penetrate clothing and obstructions, work stably under various lighting conditions, and do not record the user's facial features, better protecting user privacy. These make millimeter-wave radars an ideal choice for enhancing virtual reality semantic recognition capabilities.

[0005] Despite the advantages of mmWave radar, semantic recognition for VR presents three unique challenges. First, VR applications require real-time processing and responsiveness, which is critical to user experience and immersion. Second, existing models do not prioritize semantically relevant information that is critical to VR tasks when processing the entire mmWave point cloud data. Finally, existing methods lead to unnecessary computational overhead, memory waste, and performance inefficiency in real-time applications that require fast processing and decision making. Summary of the invention

[0006] The present invention is aimed at the problems of large computational complexity and high memory requirements in the existing millimeter wave point cloud model in virtual reality semantic recognition, and provides a virtual reality spatiotemporal data compression and semantic enhancement method based on millimeter wave point cloud, which includes at least a positioning stage and a focusing stage. In the positioning stage, the point cloud transformer assigns an attention score to each point, groups adjacent points in the point cloud, and calculates the comprehensive score of the entire group. When the semantic resolution is unclear, the neighborhood global semantic attention (NGSA) mechanism searches for areas with high semantic scores, analyzes the original point cloud data for early recognition, and reuses features using intelligent strategies; the attention mechanism is applied to the key point cloud areas selected in the positioning stage for focused recognition, determines the importance of the points in the area, and removes points that are not semantically relevant. The method of the present invention optimizes accuracy and computing resources, and at the same time uses the point cloud transformer architecture to achieve efficient data processing and feature reuse.

[0007] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud, which at least includes a positioning stage and a focusing stage.

[0008] In the localization stage, the point cloud transformer assigns an attention score to each point, groups adjacent points in the point cloud, and calculates the comprehensive score of the entire group. When the semantic resolution is unclear, the neighborhood global semantic attention (NGSA) mechanism searches for areas with high semantic scores, analyzes the original point cloud data for early recognition, and reuses features using intelligent strategies;

[0009] The focusing stage: applies the attention mechanism to the key point cloud area selected in the positioning stage for focusing and identifying, determines the importance of the points in the area and removes points that are not semantically relevant.

[0010] As an improvement of the present invention, a method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud includes the following steps:

[0011] S1: Use the point cloud transformer to process multiple frames of millimeter wave point clouds and output an attention score for each point;

[0012] S2: In the positioning stage, the point cloud data generated by the VR user's body is analyzed, the attention score of each point is calculated using the vector attention mechanism, and the points close in space are aggregated into groups. The global semantic attention score of each group is calculated to determine the key semantically distinguished areas;

[0013] S3: Introducing the vector attention mechanism, which operates on a single feature channel level and uses different feature transformations to extract feature representations for each point;

[0014] S4: When the prediction result of the localization stage is lower than the decision boundary, the focusing stage is triggered, and the neighborhood global semantic attention (NGSA) mechanism is used to identify the category-distinguishing area, select the top K points with the highest global attention score, and connect the representations of these points to construct a point cloud representation for semantic recognition;

[0015] S5: Introduce application type recognition and keyboard input recognition modules for virtual reality semantic recognition to achieve semantic enhancement of virtual reality applications based on millimeter wave point cloud spatiotemporal data compression.

[0016] As another improvement of the present invention, in step S1, the input of the point cloud transformer model is a tensor of size s×N×d, where s is the combined batch and length size, N is the number of points per frame, d is the dimension of each point, and the output is a tensor of size s×N×d attention Tensor, d attention Represents the dimension of the output feature vector for each point; the model consists of several layers that apply vector attention to the input points. The vector attention is calculated in each layer by the formula Calculated, among which is the output feature vector of the i-th point, is the attention weight between points i and j, that is, the attention score of each point; attention weight By feature transformation and ψ, relationship function β, mapping function γ (usually MLP), learning bias δ ​​and nonlinear activation function ρ are calculated as follows: shown.

[0017] As another improvement of the present invention, in step S2, the global semantic attention score of each group is specifically:

[0018]

[0019] Among them, G K represents the global semantic attention score of the K-th group.

[0020] As another improvement of the present invention, after obtaining the vector attention score of each point in the point cloud, the points are grouped according to their physical proximity and the formula is used. Assign a global attention score g to each group j , to reflect its relevance to the target semantics, where G j is the global semantic attention score of the jth group, |G j | is the point group G j The number of midpoints, x i is the point group G j The i-th point in Yes The vector attention score, is a learnable weight vector, is a vector With the weight vector The inner product of .

[0021] As another improvement of the present invention, in step S4, each group has a corresponding representation Construct a point cloud representation by concatenating the representations of the first K points

[0022]

[0023] Among them, h 1 represents the representation vector of each point group, and Top-K(G) represents the set of the top K points with the highest scores, which can be formally expressed as:

[0024]

[0025] Among them, |K′|=K, G=g 1 ,g 2 ,...,g M Represents a collection of points.

[0026] Compared with the prior art, the present invention has the following technical effects and advantages:

[0027] (1) The present invention includes at least a positioning stage and a focusing stage. When processing sparse point cloud data, it overcomes the challenges of how to focus on the moving parts of the target, especially the semantically distinguished areas, and how to use the point cloud data of these key parts to enhance virtual reality semantic recognition, thereby achieving optimization of accuracy and computing resources.

[0028] (2) In the positioning stage, the method of the present invention analyzes the original point cloud data for early recognition and reuses features using intelligent strategies to further reduce computing resource consumption. In the focusing stage, the attention mechanism is applied to the key point cloud area selected in the positioning stage, which significantly narrows the focus on the key area on the virtual reality controller, greatly reduces the computing cost of the subsequent stages, and eliminates noise interference in non-critical areas, further reducing the computing and memory resource consumption of the model.

[0029] (3) The present invention makes full use of the characteristics of perceived point cloud data to optimize the semantic recognition process; and utilizes the point cloud transformer architecture to achieve efficient data processing and feature reuse. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the steps of the method of the present invention;

[0031] Figure 2It is a working principle diagram of the application type identification module in the system of the present invention;

[0032] Figure 3 It is a working principle diagram of the keyboard input recognition module in the system of the present invention;

[0033] Figure 4 It is a system scenario and device prototype diagram in the test example of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0035] Example 1

[0036] A method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud, such as Figure 1 As shown, the following steps are included:

[0037] Step S1: Use the point cloud transformer to process multiple frames of millimeter wave point clouds and output an attention score for each point; the input of the point cloud transformer model is a tensor of size s×N×d, where s is the combined batch and length size, N is the number of points per frame, and d is the dimension of each point, containing five features: x, y, z, speed, and intensity. The output is a tensor of size s×N×d attention Tensor, d attention represents the dimension of the output feature vector for each point. The model consists of several layers that apply vector attention to the input points. The vector attention is calculated in each layer by the formula Calculated, among which is the output feature vector of the i-th point, is the attention weight between points i and j. Attention weight By feature transformation and ψ, relationship function β, mapping function γ (usually MLP), learning bias δ ​​and nonlinear activation function ρ are calculated as follows: shown.

[0038] Step S2: In the positioning stage, the point cloud data generated by the VR user's body is analyzed, the attention score of each point is calculated using the vector attention mechanism, and the points close in space are aggregated into groups, and the global semantic attention score of each group is calculated to determine the key semantic distinction area;

[0039] After obtaining the vector attention score for each point in the point cloud, the points are grouped according to their physical proximity and the formula is used Calculate the global attention score G for each group n, to reflect its relevance to the target semantics, where G n represents the nth point group. The cumulative score of each group represents the Neighborhood Global Semantic Attention (NGSA). The group with the highest NGSA score is regarded as the main semantically discriminative region and is highlighted in the point cloud. If the score is lower, continue to locate the semantically discriminative region for more precise recognition.

[0040] Step S3: By introducing the vector attention mechanism, the inherent irregularity and disorder of point cloud data are solved. Vector attention operates at the level of a single feature channel, providing significant advantages in processing the unstructured characteristics of point cloud data. The vector attention mechanism uses different feature transformations to extract the feature representation of each point and processes it through relationship functions, mapping functions, and nonlinear activation functions;

[0041] By introducing a vector attention mechanism to identify semantically distinguished regions, the inherent irregularity and disorder of point cloud data can be solved. Vector attention operates at the level of a single feature channel, providing a significant advantage in handling the unstructured nature of point cloud data. After obtaining the vector attention score for each point in the point cloud, the points are grouped according to their physical proximity and the formula is used. Assign a global attention score g to each group j , to reflect its relevance to the target semantics, where G j is the jth point group, Yes The vector attention score, is a learnable weight vector. To identify semantically distinguishable regions, we propose a novel semantic enhancement mechanism that selects the points that are most informative for the semantic recognition task by selecting the group with the highest global attention score, as shown in formula As shown, R S represents the semantically distinguishable region. This formula selects the group g that maximizes the global attention score j , that is, the area with the highest semantic relevance to the target.

[0042] Step S4: When the prediction result of the positioning stage is lower than the decision boundary, the focusing stage is triggered to further refine the process. This stage uses the NGSA mechanism to identify the category-distinguishing area, selects the top K points with the highest global attention score, and connects the representations of these points to construct a point cloud representation for semantic recognition;

[0043] Prediction result R in the positioning stage S <η, where η is the decision boundary for further localization and focus identification, triggering the focus phase to further refine the process. This phase involves using the formula The NGSA mechanism defined identifies the category-differentiated regions. The NGSA mechanism is derived from the set of all possible points G = g 1 ,g 2 ,...,g M Select the top K points with the highest global attention scores from where M is the total number of points. Each group has a corresponding representation Construct a point cloud representation by concatenating the representations of the first K points As shown in the formula, where |K′|=K, for all g j ∈K′ has s(g i )≥s(g j ), and for all g j ∈G\K′ has s(g i )>s(g j ).so The formula can be rewritten as

[0044] Step S5: To demonstrate the flexibility and reusability of the method, application type recognition and keyboard input recognition modules are introduced for virtual reality semantic recognition. The application type recognition module uses the point cloud data collected during the focusing phase to identify the application type, while the keyboard input recognition module detects keyboard input from the millimeter wave point cloud collected from the user's head display and controller.

[0045] like Figure 2 As shown in the figure, the application type recognition module uses the point cloud data of size s×K×d collected in the focusing stage to identify the application type. The feature network extracts body features of s×96 dimensions, which are then fed into the long short-term memory network module to extract continuous VR actions. Finally, the decision module classifies the output of the action module into five categories of applications. For example, game applications usually involve more body movements, while browsing is characterized by hand movements.

[0046] like Figure 3 As shown, the size of the data collected by the keyboard input recognition module from the user's head display and controller is s×(n 1 +n 2 )×d mmWave point cloud, where n 1 and n 2 It is the point cloud on the headset and controller. The feature network extracts s×64-dimensional hand features, and then feeds these hand features into the recognition network composed of a bidirectional long short-term memory network. The model is good at recognizing context-related temporal features and accurately recognizes keyboard input.

[0047] Test Case

[0048] In order to verify the effectiveness of the method proposed in this scheme, millimeter-wave point cloud data set collection, model training and testing experiments were carried out in virtual reality semantic recognition tasks to demonstrate the performance advantages and impact of this scheme.

[0049] Take the following configuration to perform the experiment:

[0050] Hardware setup: 1. Millimeter-wave radar IWR6843ISK-ODS, used to collect millimeter-wave point cloud data;

[0051] 2. Azure Kinect v2, used to collect real virtual reality information and train the network; 3. Virtual reality equipment, controllers and head displays, etc.

[0052] Experimental design: 1. The subjects were asked to wear Figure 4 The virtual reality device shown in the figure performs the specified action; 2. The actions include the following 5 types of virtual reality applications: games, browsing, social, education and entertainment; and 29 types of handle keyboard input actions;

[0053] 3. Top-1 Acc. is used to measure the accuracy of semantic recognition effect, FLOPs measures the computational complexity, and Params measures the memory usage.

[0054] We recruited 12 participants in the experiment, including 6 males and 6 females, aged between 21 and 58 years old. Before data collection, we obtained informed consent from each participant. We collected 3,600 datasets, including 12TB of millimeter-wave point cloud and Kinect data, with 100 datasets for each keystroke category. The millimeter-wave radar data contained in each dataset includes 30 seconds of raw signal and point cloud data, with a sampling rate of 10 frames. The experimental results are shown in the following table:

[0055] Table 1 Ablation experiment

[0056]

[0057] As can be seen from the table, the present invention achieves the best performance in terms of accuracy and computational economy when all design features such as attention score, group number, highest group sum, and first K points are combined. While maintaining a high accuracy of 97.6%, the amount of calculation is reduced to 1.5G and the number of parameters is reduced to 986K, which proves that the present invention greatly reduces data redundancy, improves accuracy, and reduces unnecessary complexity by effectively utilizing the rich semantic information contained in point cloud data.

[0058] It can be seen from the above experiments or data that in virtual reality semantic recognition, the present invention can achieve high accuracy and high efficiency semantic recognition. The present invention proposes a virtual reality spatiotemporal data compression and semantic enhancement method based on millimeter wave point cloud. The method uses a two-stage semantic recognition framework to overcome the challenges of high computation and memory requirements of existing millimeter wave point cloud models, and achieves efficient and accurate virtual reality semantic recognition, which is easier to implement. At the same time, compared with traditional virtual reality semantic recognition, the method of the present invention only uses low-cost radar and virtual reality equipment, does not rely on lighting, ignores harsh conditions such as smoke and dust, and avoids privacy issues.

[0059] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud, characterized by: At least including the positioning stage and the focusing stage, In the localization stage, the point cloud transformer assigns an attention score to each point, groups adjacent points in the point cloud, and calculates the comprehensive score of the entire group. When the semantic resolution is unclear, the neighborhood global semantic attention (NGSA) mechanism searches for areas with high semantic scores, analyzes the original point cloud data for early recognition, and reuses features using intelligent strategies; The focusing stage: applies the attention mechanism to the key point cloud area selected in the positioning stage for focusing and identifying, determines the importance of the points in the area and removes points that are not semantically relevant.

2. The method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud according to claim 1, characterized in that: The steps include: S1: Use the point cloud transformer to process multiple frames of millimeter wave point clouds and output an attention score for each point; S2: In the positioning stage, the point cloud data generated by the VR user's body is analyzed, the attention score of each point is calculated using the vector attention mechanism, and the points close in space are aggregated into groups. The global semantic attention score of each group is calculated to determine the key semantically distinguished areas; S3: Introducing the vector attention mechanism, which operates on a single feature channel level and uses different feature transformations to extract feature representations for each point; S4: When the prediction result of the localization stage is lower than the decision boundary, the focusing stage is triggered, and the neighborhood global semantic attention mechanism (NGSA) is used to identify the category-distinguishing area, select the top K points with the highest global attention score, and connect the representations of these points to construct a point cloud representation for semantic recognition; S5: Introduce application type recognition and keyboard input recognition modules for virtual reality semantic recognition to achieve semantic enhancement of virtual reality applications based on millimeter wave point cloud spatiotemporal data compression.

3. The method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud according to claim 2, characterized in that: In step S1, the input of the point cloud transformer model is a tensor of size s×N×d, where s is the combined batch and length size, N is the number of points per frame, and d is the dimension of each point. The output is a tensor of size s×N×d attention Tensor, d attention Represents the dimension of the output feature vector for each point; the model consists of several layers that apply vector attention to the input points. The vector attention is calculated in each layer by the formula Calculated, among which is the output feature vector of the i-th point, is the attention weight between points i and j, that is, the attention score of each point; attention weight By feature transformation The calculation results are:

4. The method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud according to claim 3, characterized in that: In step S2, the global semantic attention score of each group is specifically: Among them, G K represents the global semantic attention score of the K-th group.

5. The method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud according to claim 4, characterized in that: In step S3, after obtaining the vector attention score of each point in the point cloud, the points are grouped according to their physical proximity and the formula is used. Assign a global attention score g to each group j , where G j is the global semantic attention score of the jth group, |G j | is the point group G j The number of midpoints, x i is the point group G j The i-th point in Yes The vector attention score, is a learnable weight vector, is a vector With the weight vector The inner product of .

6. The method for virtual reality spatiotemporal data compression and semantic enhancement based on millimeter wave point cloud according to claim 4, characterized in that: In step S4, each group has a corresponding representation Construct a point cloud representation by concatenating the representations of the first K points Among them, h i represents the representation vector of each point group, and Top-K(G) represents the set of the top K points with the highest scores, which can be formally expressed as: Where |K′|=K, G=g1,g2,...,g M Represents a collection of points.

Citation Information

Patent Citations

  • A human posture regression method and system based on point cloud semantic enhancement

    CN109086683A

  • Transform-based 3D point cloud analysis method and device for enhancing local semantic learning ability

    CN115294563A

  • Urban scene laser LiDAR point cloud semantic segmentation method based on graph convolution

    CN116129118A

  • Building foot point identification and segmentation method based on two-stage 3D point cloud semantic segmentation

    CN116152494A

  • Point cloud semantic segmentation method based on global feature enhancement

    CN118247511A