Multi-scene VR interaction method

Abstract

The invention discloses a multi-scene VR (virtual reality) interaction method, which belongs to the technical field of image processing, and comprises the following steps of: cutting a gesture part of each frame of image, extracting an image of the gesture part, constructing a data feature vector of pixel points on a contour according to a contour texture feature vector and a pixel texture feature vector, and reflecting the features of the pixel points on the contour, so as to obtain a multi-scene VR interaction result. According to the method, gesture contour images are obtained, so that similar target contours are found on the gesture contour images, target points on each gesture contour image are determined, and the target points corresponding to the similar target contours on the gesture contour images are constructed into a motion track set of the target points; and the multi-channel trajectory classification model is adopted to process the motion trajectory set of the multiple target points to obtain the gesture type, so that the classification precision of the user gestures is improved, smooth VR scene switching is realized, and the interaction experience of the user in the multi-scene VR environment is greatly enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-scene VR interaction method. Background Art

[0002] Virtual reality (VR) technology has made significant progress in recent years, but it still faces many challenges in achieving natural and intuitive human-computer interaction. At present, most VR systems rely mainly on handheld controllers for interaction. Although this method is rich in functions, it limits the user's freedom and immersion to a certain extent. Some existing gesture recognition technologies based on computer vision have made it possible for controller-free interaction. Neural networks are currently used to recognize gestures, but the gesture habits and performances of different users vary greatly. Therefore, a single gesture recognition cannot determine user needs, and there is a problem of low accuracy in classifying user gestures. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, a multi-scenario VR interaction method provided by the present invention solves the existing problem of low accuracy in user gesture classification.

[0004] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a multi-scene VR interaction method, comprising the following steps:

[0005] S1, cutting the gesture of each frame image in the operator's action video to obtain a gesture image;

[0006] S2, extracting the contour of each gesture image, and constructing the data feature vector of the pixel points on the contour based on the contour texture feature vector and the pixel texture feature vector;

[0007] S3, according to the data feature vector of the pixel points on the contour, searching for similar target contours for multiple gesture contour images, and constructing a set of motion trajectories of the target points;

[0008] S4, obtaining the gesture type based on the motion trajectory set of multiple target points and the multi-channel trajectory classification model;

[0009] S5. Switch VR scenes according to the gesture type.

[0010] Furthermore, the S1 comprises the following sub-steps:

[0011] S11, collecting action video of the operator;

[0012] S12, using the YOLO neural network to recognize the gesture of each frame image in the action video and obtain the gesture target frame;

[0013] S13. Cut each frame of image according to the gesture target frame of each frame of image to obtain a gesture image.

[0014] Furthermore, the S2 comprises the following sub-steps:

[0015] S21, extracting the contour of each gesture image to obtain a gesture contour image;

[0016] S22, constructing a contour texture feature vector for each pixel in the gesture contour image;

[0017] S23, constructing a pixel texture feature vector for each pixel in the gesture image;

[0018] S34, combining the contour texture feature vector and the pixel texture feature vector of the same pixel point on the gesture contour image to obtain a data feature vector of the pixel point on the contour.

[0019] Further, the elements of the contour texture feature vector in S22 include: curvature and curvature variation coefficient;

[0020] The expression of curvature variation coefficient is: , where μ is the curvature variation coefficient, e is a natural constant, and θ i is the curvature of the i-th pixel in the pixel neighborhood of the gesture contour image, N is the number of pixels in the pixel neighborhood of the gesture contour image, and i is a positive integer.

[0021] Further, the pixel texture feature vector in S23 includes: a pixel mean value and a pixel uneven value;

[0022] Among them, the pixel mean is the mean of the pixel values ​​within the neighborhood of the pixel point;

[0023] The expression of pixel non-uniform value is: , where γ is the pixel unevenness value, e is a natural constant, and f max is the maximum pixel value in the neighborhood of the pixel point in the gesture image, f min It is the minimum pixel value in the neighborhood of the pixel point in the gesture image.

[0024] Furthermore, S3 includes the following sub-steps:

[0025] S31, taking a pixel point in the gesture contour image as a landmark point;

[0026] S32, calculating the similarity between the landmark point and the pixel points in the remaining plurality of gesture contour images on the data feature vector;

[0027] S33, when multiple similarities are greater than the threshold, extracting pixel points similar to the marker points in the remaining gesture contour image as points to be compared;

[0028] S34, calculating the similarity between the pixel points in the neighborhood range of the point to be compared and the pixel points in the neighborhood range of the marker point on the data feature vector, when the similarities are all greater than the threshold value, the marker point and the pixel points in the neighborhood range of the marker point and the point to be compared and the pixel points in the neighborhood range of the point to be compared are similar target contours;

[0029] S35, extracting the target contour on each gesture contour image to obtain the target point coordinates;

[0030] S36 , the target point coordinates corresponding to similar target contours on the plurality of gesture contour images are used to form a set of motion trajectories of the target points.

[0031] Furthermore, the calculation formula of the target point coordinates in S35 is: , , where x o is the horizontal coordinate of the target point, y o is the ordinate of the target point, x j is the horizontal coordinate of the jth pixel point in the target contour, y j is the ordinate of the jth pixel in the target contour, j is a positive integer, and M is the number of pixels in similar target contours.

[0032] Further, the multi-channel trajectory classification model in S4 includes: a plurality of motion trajectory feature extraction units, a plurality of fully connected layers, a clustering layer and an output layer;

[0033] The input end of each motion trajectory feature extraction unit is used to input a motion trajectory set of a target point; the input end of each fully connected layer is connected to the output end of a motion trajectory feature extraction unit, and its output end is connected to the input end of the clustering layer; the input end of the output layer is connected to the output end of the clustering layer, and its output end serves as the output end of the multi-channel trajectory classification model; the clustering layer is used to cluster the classification values ​​output by the fully connected layer to find the largest class; the output layer is used to weight the classification values ​​in the largest class to obtain the final classification value.

[0034] Furthermore, each of the motion trajectory feature extraction units includes: a horizontal coordinate perception LSTM layer, a vertical coordinate perception LSTM layer, a coordinate fusion layer, an attention application layer, a BN layer and a convolution layer;

[0035] The abscissa-sensing LSTM layer is used to input a abscissa in a motion trajectory set of a target point, wherein one cell unit in the abscissa-sensing LSTM layer processes one abscissa correspondingly;

[0036] The ordinate perception LSTM layer is used to input the ordinate in the motion trajectory set of a target point, wherein one cell unit in the ordinate perception LSTM layer processes one ordinate correspondingly;

[0037] The input end of the coordinate fusion layer is connected to the output end of the horizontal coordinate perception LSTM layer and the output end of the vertical coordinate perception LSTM layer respectively, and its output end is connected to the input end of the attention application layer; the output end of the attention application layer is connected to the input end of the BN layer; the input end of the convolution layer is connected to the output end of the BN layer, and its output end serves as the output end of the motion trajectory feature extraction unit.

[0038] Furthermore, the expression of the coordinate fusion layer is: , where y k is the kth fusion feature value output by the coordinate fusion layer, h x,k is the kth abscissa eigenvalue output by the abscissa-aware LSTM layer, h y,k is the kth ordinate eigenvalue output by the ordinate-aware LSTM layer, ω x,k h x,k The weight, ω y,k h y,k The weight of , Sigmoid is the activation function, and k is a positive integer;

[0039] The expression of the attention application layer is: , where r k is the kth eigenvalue output by the attention application layer, and K is the number of fused eigenvalues ​​output by the coordinate fusion layer.

[0040] The beneficial effects of the present invention are as follows: the present invention cuts the gesture part of each frame image, extracts the image of the gesture part, constructs the data feature vector of the pixel points on the contour according to the contour texture feature vector and the pixel texture feature vector, reflects the characteristics of the pixel points on the contour, thereby finding similar target contours on each gesture contour image, thereby determining the target points on each gesture contour image, constructing the target points corresponding to the similar target contours on each gesture contour image as a set of motion trajectories of the target points, adopts a multi-channel trajectory classification model to process the motion trajectory set of multiple target points, obtains the gesture type, improves the classification accuracy of the user's gesture, realizes smooth VR scene switching, and greatly enhances the user's interactive experience in a multi-scene VR environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a multi-scene VR interaction method;

[0042] Figure 2 It is a structural diagram of the multi-channel trajectory classification model;

[0043] Figure 3 Schematic diagram of the structure of the motion trajectory feature extraction unit. DETAILED DESCRIPTION

[0044] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0045] like Figure 1 As shown, a multi-scene VR interaction method includes the following steps:

[0046] S1, cutting the gesture of each frame image in the operator's action video to obtain a gesture image;

[0047] S2, extracting the contour of each gesture image, and constructing the data feature vector of the pixel points on the contour based on the contour texture feature vector and the pixel texture feature vector;

[0048] S3, according to the data feature vector of the pixel points on the contour, searching for similar target contours for multiple gesture contour images, and constructing a set of motion trajectories of the target points;

[0049] S4, obtaining the gesture type based on the motion trajectory set of multiple target points and the multi-channel trajectory classification model;

[0050] S5. Switch VR scenes according to the gesture type.

[0051] The S1 comprises the following sub-steps:

[0052] S11, collecting action video of the operator;

[0053] S12, using the YOLO neural network to recognize the gesture of each frame image in the action video and obtain the gesture target frame;

[0054] S13. Cut each frame of image according to the gesture target frame of each frame of image to obtain a gesture image.

[0055] The S2 comprises the following sub-steps:

[0056] S21, extracting the contour of each gesture image to obtain a gesture contour image;

[0057] S22, constructing a contour texture feature vector for each pixel in the gesture contour image;

[0058] S23, constructing a pixel texture feature vector for each pixel in the gesture image;

[0059] S34, combining the contour texture feature vector and the pixel texture feature vector of the same pixel point on the gesture contour image to obtain a data feature vector of the pixel point on the contour.

[0060] The present invention extracts the contour of each gesture image to obtain a gesture contour image, constructs a contour texture feature vector for each pixel point on the gesture contour image to reflect the contour feature, constructs a pixel texture feature vector for each pixel point in the gesture image to reflect the pixel value distribution feature, and constructs a data feature vector only for the pixel points on the gesture contour image to facilitate contour comparison. The contour texture feature vector of the same pixel point on the gesture contour image is derived from the gesture contour image, and the pixel texture feature vector is derived from the gesture image (in the gesture contour image, there is only the image contour and a large amount of pixel value information is lost, so the pixel texture feature vector is derived from the gesture image to fully reflect the pixel value distribution in the local area).

[0061] The elements of the contour texture feature vector in S22 include: curvature and curvature variation coefficient;

[0062] The expression of curvature variation coefficient is: , where μ is the curvature variation coefficient, e is a natural constant, and θ i is the curvature of the i-th pixel in the pixel neighborhood of the gesture contour image, N is the number of pixels in the pixel neighborhood of the gesture contour image, and i is a positive integer.

[0063] The present invention uses the curvature and curvature variation coefficient at the pixel point as the contour texture features of the pixel point. The curvature directly reflects the degree of curvature of the gesture contour at each point. A high curvature value indicates that the contour is sharply curved at the point, while a low curvature value indicates that the contour is relatively straight at the point. The curvature variation coefficient reflects the rate of change of the contour curvature in the local area, and describes the degree of change of the curvature from one point to the adjacent point. It helps to capture the subtle changes and turning points of the gesture contour, and sets an exponential function to further highlight the turning points.

[0064] The pixel texture feature vector in S23 includes: a pixel mean value and a pixel uneven value;

[0065] Among them, the pixel mean is the mean of the pixel values ​​within the neighborhood of the pixel point;

[0066] The expression of pixel non-uniform value is: , where γ is the pixel unevenness value, e is a natural constant, and f max is the maximum pixel value in the neighborhood of the pixel point in the gesture image, f min It is the minimum pixel value in the neighborhood of the pixel point in the gesture image.

[0067] The present invention reflects the imaging characteristics of the pixel point in the local area through the pixel mean, reflects the distribution of the pixel value in the local area through the pixel uneven value, and introduces nonlinear mapping by using the exponential function to calculate the uneven value, so as to be more sensitive to slight texture changes and help capture subtle features such as fingerprints and finger joints.

[0068] The S3 comprises the following sub-steps:

[0069] S31, taking a pixel point in the gesture contour image as a landmark point;

[0070] S32, calculating the similarity between the landmark point and the pixel points in the remaining plurality of gesture contour images on the data feature vector;

[0071] S33, when multiple similarities are greater than the threshold, extracting pixel points similar to the marker points in the remaining gesture contour image as points to be compared;

[0072] In the present invention, similarity means that the similarity is greater than a threshold;

[0073] S34, calculating the similarity between the pixel points in the neighborhood range of the point to be compared and the pixel points in the neighborhood range of the marker point on the data feature vector, when the similarities are all greater than the threshold value, the marker point and the pixel points in the neighborhood range of the marker point and the point to be compared and the pixel points in the neighborhood range of the point to be compared are similar target contours;

[0074] S35, extracting the target contour on each gesture contour image to obtain the target point coordinates;

[0075] S36 , the target point coordinates corresponding to similar target contours on the plurality of gesture contour images are used to form a set of motion trajectories of the target points.

[0076] The present invention selects pixel points in the gesture contour image as marker points one by one, and screens out target points corresponding to target contours that meet S33 and S44. After performing a single pixel comparison in step S32, S34 considers the similarity of pixels in the neighborhood of the pixel point. When the similarities are greater than a threshold, the contour where the marker point and the pixels in the neighborhood of the marker point are located are similar to the contour where the point to be compared and the pixels in the neighborhood of the point to be compared are located.

[0077] In the present invention, a motion trajectory set of a target point is composed of target point coordinates corresponding to similar target contours on multiple gesture contour images (for a motion trajectory set of a target point, there is only one similar target contour in a gesture contour image). Since S31~S36 can be repeated multiple times, a motion trajectory set of multiple target points can be obtained, which reflects the movement process of the gesture.

[0078] The calculation formula of the target point coordinates in S35 is: , , where x o is the horizontal coordinate of the target point, y o is the ordinate of the target point, x j is the horizontal coordinate of the jth pixel point in the target contour, y j is the ordinate of the jth pixel in the target contour, j is a positive integer, and M is the number of pixels in similar target contours.

[0079] like Figure 2 As shown, the multi-channel trajectory classification model in S4 includes: multiple motion trajectory feature extraction units, multiple fully connected layers, a clustering layer and an output layer;

[0080] The input end of each motion trajectory feature extraction unit is used to input a motion trajectory set of a target point; the input end of each fully connected layer is connected to the output end of a motion trajectory feature extraction unit, and its output end is connected to the input end of the clustering layer; the input end of the output layer is connected to the output end of the clustering layer, and its output end serves as the output end of the multi-channel trajectory classification model; the clustering layer is used to cluster the classification values ​​output by the fully connected layer to find the largest class; the output layer is used to weight the classification values ​​in the largest class to obtain the final classification value.

[0081] In this embodiment, the expression of the output layer is: , where C is the final classification value, c n is the nth classification value in the largest class, L is the number of classification values ​​in the largest class, and n is a positive integer.

[0082] In this embodiment, the clustering layer uses a clustering algorithm to classify classification values ​​with similar distances into one category.

[0083] The present invention processes the motion trajectory of each target point separately. Each fully connected layer outputs a classification value. The largest class is found through clustering processing, which effectively reduces the influence of outliers and improves the stability and reliability of classification. The output layer weights the classification values ​​in the largest class, which can more accurately reflect the importance of each classification value and improve the accuracy of the final classification result.

[0084] like Figure 3 As shown, each of the motion trajectory feature extraction units includes: a horizontal coordinate perception LSTM layer, a vertical coordinate perception LSTM layer, a coordinate fusion layer, an attention application layer, a BN layer and a convolution layer;

[0085] The abscissa-sensing LSTM layer is used to input a abscissa in a motion trajectory set of a target point, wherein one cell unit in the abscissa-sensing LSTM layer processes one abscissa correspondingly;

[0086] The ordinate perception LSTM layer is used to input the ordinate in the motion trajectory set of a target point, wherein one cell unit in the ordinate perception LSTM layer processes one ordinate correspondingly;

[0087] The input end of the coordinate fusion layer is connected to the output end of the horizontal coordinate perception LSTM layer and the output end of the vertical coordinate perception LSTM layer respectively, and its output end is connected to the input end of the attention application layer; the output end of the attention application layer is connected to the input end of the BN layer; the input end of the convolution layer is connected to the output end of the BN layer, and its output end serves as the output end of the motion trajectory feature extraction unit.

[0088] The present invention uses the horizontal coordinate perception LSTM layer and the vertical coordinate perception LSTM layer respectively. The model can independently capture the temporal characteristics of the horizontal and vertical coordinates, and better understand the spatial structure and dynamic changes of the gesture trajectory. The coordinate fusion layer fuses the features of the horizontal and vertical coordinates, comprehensively considers the trajectory information in the two-dimensional space, and provides a more comprehensive feature representation. The attention application layer introduces the attention mechanism, which can highlight the key points in the trajectory and improve the model's sensitivity to important features. The use of the Batch Normalization layer helps to stabilize the training process, accelerate convergence, and improve the generalization ability of the model. The convolution layer can effectively extract local spatial features and capture the detailed information of the gesture trajectory. The convolution kernel size of the convolution layer is .

[0089] In this embodiment, both the horizontal coordinate sensing LSTM layer and the vertical coordinate sensing LSTM layer are single-layer LSTM layers.

[0090] The expression of the coordinate fusion layer is: , where y k is the kth fusion feature value output by the coordinate fusion layer, h x,k is the kth abscissa eigenvalue output by the abscissa-aware LSTM layer, h y,k is the kth ordinate eigenvalue output by the ordinate-aware LSTM layer, ω x,k h x,k The weight, ω y,k h y,k The weight of , Sigmoid is the activation function, and k is a positive integer.

[0091] The expression of the attention application layer is: , where r k is the kth eigenvalue output by the attention application layer, and K is the number of fused eigenvalues ​​output by the coordinate fusion layer.

[0092] The coordinate fusion layer uses the Sigmoid activation function to achieve nonlinear fusion of horizontal and vertical coordinate features, which can capture more complex feature relationships. The attention application layer dynamically enhances the influence of important features according to the size of the fusion feature value output by the coordinate fusion layer.

[0093] In this embodiment, the classification value 0.1-0.3 is a pointing gesture, scene switching: the user points to a scene entrance (such as a door or a portal), and the system will teleport the user to the new scene.

[0094] In this embodiment, the classification value 0.3-0.5 is a grabbing gesture, scene switching: the user grabs and pulls a specific environmental element (such as opening a curtain), which triggers a scene change (such as switching from an indoor view to an outdoor view).

[0095] In this embodiment, the classification value 0.5-0.7 is a waving gesture. Waving to the left may trigger a scene switch to the left, while waving to the right may trigger a scene switch to the right, achieving a scene transition similar to a page turning effect.

[0096] In this embodiment, the classification value 0.7-0.9 is a zoom gesture, scene switching: the user can "go deeper" into a microscopic scene (such as switching from a macroscopic galaxy view to the surface of a specific planet) through a zoom-in gesture, or "zoom out" the perspective (such as switching from the surface of the earth to a space view) through a zoom-out gesture.

[0097] In this embodiment, the classification value 0.9-1.0 is a rotation gesture, scene switching: the user can change the viewing angle through a rotation gesture, such as rotating the viewing angle in a panoramic scene, or switching views of different dimensions in a multi-dimensional scene (such as switching from a front view to a side or top view).

[0098] The present invention cuts the gesture part of each frame image, extracts the image of the gesture part, constructs the data feature vector of the pixel points on the contour according to the contour texture feature vector and the pixel texture feature vector, reflects the characteristics of the pixel points on the contour, and thus finds similar target contours on each gesture contour image, thereby determining the target points on each gesture contour image, constructs the target points corresponding to the similar target contours on each gesture contour image as a motion trajectory set of the target points, uses a multi-channel trajectory classification model to process the motion trajectory set of multiple target points, obtains the gesture type, improves the classification accuracy of the user's gesture, realizes smooth VR scene switching, and greatly enhances the user's interactive experience in a multi-scene VR environment.

[0099] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-scene VR interaction method, characterized in that: The following steps are involved: S1, cutting the gesture of each frame image in the operator's action video to obtain a gesture image; S2, extracting the contour of each gesture image, and constructing the data feature vector of the pixel points on the contour based on the contour texture feature vector and the pixel texture feature vector; S3, searching for the target contour for multiple gesture contour images according to the data feature vector of the pixel points on the contour, and constructing a set of motion trajectories of the target points; S4, obtaining the gesture type based on the motion trajectory set of multiple target points and the multi-channel trajectory classification model; S5. Switch VR scenes according to the gesture type.

2. The multi-scene VR interaction method according to claim 1, characterized in that: The S1 comprises the following sub-steps: S11, collecting action video of the operator; S12, using the YOLO neural network to recognize the gesture of each frame image in the action video and obtain the gesture target frame; S13. Cut each frame of image according to the gesture target frame of each frame of image to obtain a gesture image.

3. The multi-scene VR interaction method according to claim 1, characterized in that: The S2 comprises the following sub-steps: S21, extracting the contour of each gesture image to obtain a gesture contour image; S22, constructing a contour texture feature vector for each pixel in the gesture contour image; S23, constructing a pixel texture feature vector for each pixel in the gesture image; S34, combining the contour texture feature vector and the pixel texture feature vector of the same pixel point on the gesture contour image to obtain a data feature vector of the pixel point on the contour.

4. The multi-scene VR interaction method according to claim 3, characterized in that: The elements of the contour texture feature vector in S22 include: curvature and curvature variation coefficient; The expression of curvature variation coefficient is: , where μ is the curvature variation coefficient, e is a natural constant, and θ i is the curvature of the i-th pixel in the pixel neighborhood of the gesture contour image, N is the number of pixels in the pixel neighborhood of the gesture contour image, and i is a positive integer.

5. The multi-scene VR interaction method according to claim 3, characterized in that: The pixel texture feature vector in S23 includes: a pixel mean value and a pixel uneven value; Among them, the pixel mean is the mean of the pixel values ​​within the neighborhood of the pixel point; The expression of pixel non-uniform value is: , where γ is the pixel unevenness value, e is a natural constant, and f max is the maximum pixel value in the neighborhood of the pixel point in the gesture image, f min It is the minimum pixel value in the neighborhood of the pixel point in the gesture image.

6. The multi-scene VR interaction method according to claim 1, characterized in that: The S3 comprises the following sub-steps: S31, taking a pixel point in the gesture contour image as a landmark point; S32, calculating the similarity between the landmark point and the pixel points in the remaining plurality of gesture contour images on the data feature vector; S33, when multiple similarities are greater than the threshold, the corresponding pixel points of the remaining gesture contour image are named as similar pixel points, and the pixel points similar to the marker point in the remaining gesture contour image are extracted as points to be compared; S34, calculating the similarity between the pixel points in the neighborhood range of the point to be compared and the pixel points in the neighborhood range of the marker point on the data feature vector, when the similarities are all greater than the threshold value, the marker point and the pixel points in the neighborhood range of the marker point and the point to be compared and the pixel points in the neighborhood range of the point to be compared are similar target contours; S35, extracting the target contour on each gesture contour image to obtain the target point coordinates; S36 , the target point coordinates corresponding to similar target contours on the plurality of gesture contour images are used to form a set of motion trajectories of the target points.

7. The multi-scenario VR interaction method according to claim 6, characterized in that: The calculation formula of the target point coordinates in S35 is: , , where x o is the horizontal coordinate of the target point, y o is the ordinate of the target point, x j is the horizontal coordinate of the jth pixel point in the target contour, y j is the ordinate of the jth pixel in the target contour, j is a positive integer, and M is the number of pixels in similar target contours.

8. The multi-scene VR interaction method according to claim 1, characterized in that: The multi-channel trajectory classification model in S4 includes: multiple motion trajectory feature extraction units, multiple fully connected layers, a clustering layer and an output layer; The input end of each motion trajectory feature extraction unit is used to input a motion trajectory set of a target point; the input end of each fully connected layer is connected to the output end of a motion trajectory feature extraction unit, and its output end is connected to the input end of the clustering layer; the input end of the output layer is connected to the output end of the clustering layer, and its output end serves as the output end of the multi-channel trajectory classification model; the clustering layer is used to cluster the classification values ​​output by the fully connected layer to find the largest class; the output layer is used to weight the classification values ​​in the largest class to obtain the final classification value.

9. The multi-scene VR interaction method according to claim 8, characterized in that: Each of the motion trajectory feature extraction units includes: a horizontal coordinate perception LSTM layer, a vertical coordinate perception LSTM layer, a coordinate fusion layer, an attention application layer, a BN layer and a convolution layer; The abscissa-sensing LSTM layer is used to input a abscissa in a motion trajectory set of a target point, wherein one cell unit in the abscissa-sensing LSTM layer processes one abscissa correspondingly; The ordinate perception LSTM layer is used to input the ordinate in the motion trajectory set of a target point, wherein one cell unit in the ordinate perception LSTM layer processes one ordinate correspondingly; The input end of the coordinate fusion layer is connected to the output end of the horizontal coordinate perception LSTM layer and the output end of the vertical coordinate perception LSTM layer respectively, and its output end is connected to the input end of the attention application layer; the output end of the attention application layer is connected to the input end of the BN layer; the input end of the convolution layer is connected to the output end of the BN layer, and its output end serves as the output end of the motion trajectory feature extraction unit.

10. The multi-scene VR interaction method according to claim 9, characterized in that: The expression of the coordinate fusion layer is: , where y k is the kth fusion feature value output by the coordinate fusion layer, h x,k is the kth abscissa eigenvalue output by the abscissa-aware LSTM layer, h y,k is the kth ordinate eigenvalue output by the ordinate-aware LSTM layer, ω x,k h x,k The weight, ω y,k h y,k The weight of , Sigmoid is the activation function, and k is a positive integer; The expression of the attention application layer is: , where r k is the kth eigenvalue output by the attention application layer, and K is the number of fused eigenvalues ​​output by the coordinate fusion layer.

Images