VR interactive system and method for large space scenes

Abstract

The present application provides a VR interaction system and method for large space scenes, which relates to the field of VR interaction. It uses AI-based video data processing and analysis technology to perform key frame sampling and semantic feature association extraction on the action video of the VR experiencer in the real space, and performs time series encoding on the time queue of the position data in the real space, so as to intelligently generate the target object position change decoding value according to the query matching between each action-related semantic feature and the position time series feature, and obtain the interactive feedback result based on the difference between the decoding value and the initial position data of the target object in the real space. In this way, the relative position of the target object can be adaptively adjusted to better understand the user's action intention and context information, capture the user's coherent actions and subtle action changes, and thus more finely and intelligently adjust the position of the target object to provide a more flexible interactive experience.

Description

Technical Field

[0001] The present application relates to the field of VR interaction, and more specifically, to a VR interaction system and method for large space scenes. Background Art

[0002] VR uses a computer-generated three-dimensional environment to make users feel like they are in a virtual world. This technology is widely used in games, training, medical and other fields to provide users with an immersive experience. In a VR system, users view virtual scenes through head-mounted devices (such as VR helmets) and interact with them through various sensors and controllers.

[0003] Patent CN111443812B proposes a VR-based free movement method, VR device, equipment and storage medium, which obtains the position information of objects in the real space, sets the position of the virtual character, detects the interaction with the components in the real space, adjusts the height of the virtual character, and ensures that the movement and position of the virtual character are consistent with the user's behavior in the real space.

[0004] In the above patent, when the experiencer is detected to interact with the component in the real space, the position of the virtual component is determined by calculating the relative position relationship between the experiencer and the component and performing simple mathematical operations. However, relying solely on the relative position relationship ignores the contextual information of the action. For example, when a user performs a series of continuous actions, the simple relative position information may not capture the continuity and intention of the action, which may cause the position adjustment of the virtual component (target object) to be inaccurate or unnatural, especially in complex scenes. In addition, relying solely on position data may not capture subtle changes in movement. For example, slight changes in the user's gestures may not be accurately reflected, so that the position adjustment of the virtual component is not fine enough, affecting the user experience.

[0005] Therefore, an optimized VR interaction solution for large space scenes is desired. Summary of the invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a VR interaction system and method for large space scenes, which uses AI-based video data processing and analysis technology to perform key frame sampling and semantic feature association extraction on the action video of the VR experiencer in the real space, and time-series encoding on the time queue of the position data in the real space, so as to intelligently generate the target object position change decoding value according to the query matching between each action-associated semantic feature and the position timing feature, and obtain the interactive feedback result based on the difference between the decoded value and the initial position data of the target object in the real space. In this way, the relative position of the target object can be adaptively adjusted to better understand the user's action intention and contextual information, capture the user's coherent actions and subtle action changes, and adjust the position of the target object more finely and intelligently to provide a more flexible interactive experience.

[0007] According to one aspect of the present application, a VR interaction system for large space scenes is provided, which includes: an interactive action video position data acquisition module, which is used to acquire an action video of a VR experiencer in a real space collected by a VR head-mounted device, and collect a time queue of the position data of the VR experiencer in the real space through a position sensor; an initial position data acquisition module, which is used to acquire the initial position data of a target object interacting with the VR experiencer in the real space; a position change value determination module, which is used to determine a position change decoding value of the target object based on the action video and the time queue of the position data, wherein the position change value determination module includes: an action video key frame sampling unit, which is used to perform key frame sampling on the action video to obtain a time queue of action key frames; an action semantic association unit, which is used to associate the action semantics with the action video; The time queue of the action key frame performs action semantic feature extraction and temporal context encoding to obtain a time series of context action associated semantic expression features; a position temporal encoding unit is used to perform position temporal encoding on the time queue of the position data to obtain position temporal features; a position change query matching optimization representation generation unit is used to perform local neighborhood optimized query matching on the time series of the context action associated semantic expression features and the position temporal features to obtain a position change query matching optimization representation; a position change decoding unit is used to obtain a position change decoding value of the target object based on the position change query matching optimization representation; an interactive feedback result generation module is used to subtract the target object position change decoding value from the initial position data to obtain the relative position change data of the target object as an interactive feedback result.

[0008] In the above-mentioned VR interaction system for large space scenes, the action semantic association unit includes: an action semantic feature extraction subunit, which is used to pass the time queue of the action key frames through an action semantic feature extractor based on void convolution coding to obtain a time series of action latent semantic feature vectors; an action semantic feature context encoding subunit, which is used to pass the time series of the action latent semantic feature vectors through an action timing context encoder based on a converter structure to obtain a time series of context action association semantic expression feature vectors as the time series of context action association semantic expression features.

[0009] In the above-mentioned VR interactive system for large space scenes, the position timing encoding unit is used to: arrange the time queue of the position data into an experiencer's position timing vector and then input it into a position timing encoder to obtain a position timing feature vector as the position timing feature.

[0010] In the above-mentioned VR interaction system for large space scenes, the position change query matching optimization representation generation unit includes: a position time series feature value embedding coding subunit, which is used to process the position time series feature vector using a value embedding coding matrix to obtain a position time series value feature vector; a context action associated semantic expression feature one-dimensional convolution coding subunit, which is used to perform one-dimensional convolution coding on the time series of the context action associated semantic expression feature vector to obtain a sequence of context action associated local neighborhood semantic association feature vectors; an action semantic position query fusion matching subunit, which is used to use the time series of the context action associated semantic expression feature vector and the sequence of the context action associated local neighborhood semantic association feature vector as a sequence of key vectors, and use the position time series value feature vector as a value vector and the position time series feature vector as a query vector, and perform attention-based query fusion matching on the sequence of key vectors, the value vector and the query vector to obtain a position change query matching action implicit optimization representation vector as the position change query matching optimization representation.

[0011] In the above-mentioned VR interaction system for large space scenes, the scale of the one-dimensional convolutional coding is equal to an integer multiple of the length of the context action associated semantic expression feature vector.

[0012] In the above-mentioned VR interaction system for large space scenes, the action semantic position query fusion matching subunit includes: a context action associated semantic granularity encoding secondary subunit, which is used to use the position time series feature vector as the query vector, the position time series value feature vector as the value vector and the time series of the context action associated semantic expression feature vector as the sequence of key vectors, and input the sequence of the query vector, the value vector and the key vector into a first multi-head attention query module based on a converter structure to obtain a sequence of context action associated semantic granularity query matching implicit vectors; a context action associated semantic neighborhood granularity query matching secondary subunit, which is used to use the position time series feature vector as the query vector, the position time series value feature vector as the value vector and the sequence of the context action associated local neighborhood semantic association feature vector as the sequence of key vectors, and input the query vector, the value vector and the key vector into a first multi-head attention query module based on a converter structure to obtain a sequence of context action associated semantic granularity query matching implicit vectors. and the sequence of key vectors are input into a second multi-head attention query module based on a converter structure to obtain a sequence of context-action associated semantic neighborhood granularity query matching implicit vectors; a context-action associated semantic granularity semantic neighborhood granularity vector mean calculation secondary subunit is used to calculate the sequence of context-action associated semantic granularity query matching implicit vectors and the positional mean vector of the sequence of context-action associated semantic granularity query matching implicit vectors to obtain context-action associated semantic granularity query matching implicit feature vectors and context-action associated semantic neighborhood granularity query matching implicit feature vectors; a context-action associated semantic granularity-semantic neighborhood granularity fusion secondary subunit is used to fuse the context-action associated semantic granularity query matching implicit feature vectors and the context-action associated semantic neighborhood granularity query matching implicit feature vectors to obtain the position change query matching action implicit optimization representation vector.

[0013] In the above-mentioned VR interaction system for large space scenes, the context-action association semantic granularity encoding secondary sub-unit is used to: calculate the product between the position-time series feature vector and the transposed vector of the context-action association semantic expression feature vector, divide the obtained position-context action semantic implicit association matrix by the square root of the length of the context-action association semantic expression feature vector by position to obtain the position-context action semantic implicit association modulation matrix; input the position-context action semantic implicit association modulation matrix into the softmax function to obtain the position-context action semantic implicit association activation matrix; multiply the position-context action semantic implicit association activation matrix by the position-time series value feature vector to obtain the context-action association semantic granularity query matching implicit vector.

[0014] In the above-mentioned VR interaction system for large space scenes, the context action associated semantic granularity-semantic neighborhood granularity fusion secondary subunit is used to: respectively calculate the positional subtraction, positional dot product and positional addition between the context action associated semantic granularity query matching implicit feature vector and the context action associated semantic neighborhood granularity query matching implicit feature vector to obtain the context action associated semantic granularity-semantic neighborhood granularity difference vector, the context action associated semantic granularity-semantic neighborhood granularity dot product vector and the context action associated semantic granularity-semantic neighborhood granularity sum vector; the context action associated semantic granularity-semantic neighborhood granularity The associated semantic granularity-semantic neighborhood granularity difference vector, the context action associated semantic granularity-semantic neighborhood granularity dot product vector and the context action associated semantic granularity-semantic neighborhood granularity sum vector are cascaded to obtain a context action associated semantic granularity-semantic neighborhood granularity cascade vector; after performing one-dimensional convolution encoding on the context action associated semantic granularity-semantic neighborhood granularity cascade vector, the obtained context action associated semantic granularity-semantic neighborhood granularity convolution encoding vector is subjected to local window-based maximum pooling processing to obtain the implicit optimized representation vector of the position change query matching action.

[0015] In the above-mentioned VR interactive system for large space scenes, the position change decoding unit is used to: pass the implicit optimization representation vector of the position change query matching action through a decoder-based target object position changer to obtain the target object position change decoding value.

[0016] According to another aspect of the present application, a VR interaction method for large space scenes is provided, which includes: obtaining an action video of a VR experiencer in a real space collected by a VR head-mounted device, and collecting a time queue of the position data of the VR experiencer in the real space through a position sensor; obtaining the initial position data of a target object interacting with the VR experiencer in the real space; determining a target object position change decoding value based on the action video and the time queue of the position data, including: performing key frame sampling on the action video to obtain a time queue of action key frames; performing action semantic feature extraction and temporal context encoding on the time queue of the action key frames to obtain a time series of context action-associated semantic expression features; performing position temporal encoding on the time queue of the position data to obtain a position temporal feature; performing local neighborhood optimized query matching on the time series of the context action-associated semantic expression features and the position temporal feature to obtain a position change query matching optimization representation; obtaining the target object position change decoding value based on the position change query matching optimization representation; subtracting the target object position change decoding value from the initial position data to obtain the relative position change data of the target object as the interaction feedback result.

[0017] Compared with the prior art, the VR interaction system and method for large space scenes provided by the present application adopts AI-based video data processing and analysis technology to perform key frame sampling and semantic feature association extraction on the action video of the VR experiencer in the real space, and performs time series encoding on the time queue of the position data in the real space, so as to intelligently generate the target object position change decoding value based on the query matching between each action-related semantic feature and the position time series feature, and obtain the interactive feedback result based on the difference between the decoding value and the initial position data of the target object in the real space. In this way, the relative position of the target object can be adaptively adjusted to better understand the user's action intention and contextual information, capture the user's coherent actions and subtle action changes, and thus more finely and intelligently adjust the position of the target object to provide a more flexible interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] In the attached picture: Figure 1 This is a system block diagram of a VR interaction system for large space scenes according to an embodiment of the present application.

[0020] Figure 2 This is a block diagram of a position change value determination module in a VR interaction system for a large space scene according to an embodiment of the present application.

[0021] Figure 3 This is a schematic diagram of data flow of a position change value determination module in a VR interaction system for a large space scene according to an embodiment of the present application.

[0022] Figure 4 This is a block diagram of an action semantic association unit in a VR interaction system for large space scenes according to an embodiment of the present application.

[0023] Figure 5 This is a block diagram of a position change query matching optimization representation generation unit in a VR interaction system for large space scenes according to an embodiment of the present application.

[0024] Figure 6 This is a block diagram of an action semantic position query fusion matching subunit in a VR interaction system for large space scenes according to an embodiment of the present application.

[0025] Figure 7 This is a flowchart of a VR interaction method for a large space scene according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0027] Virtual reality (VR) technology creates an experience for users as if they are in a virtual world through a computer-generated three-dimensional environment. This technology has been widely used in many fields such as games, education and training, and health care, bringing an immersive experience to users. In a VR system, users watch virtual scenes by wearing head-mounted devices (such as VR helmets) and interact with the virtual environment through various sensors and controllers.

[0028] Patent CN111443812B introduces a VR-based free movement method, VR device, equipment and storage medium, which captures the position information of objects in the real space, sets the position of the virtual character, detects the interaction between the user and the components in the real space, adjusts the height of the virtual character, and ensures that the movement and position of the virtual character are synchronized with the user's behavior in the real space.

[0029] However, in the above patent, when the user is detected to interact with the real space component, it mainly relies on calculating the relative position relationship between the user and the component, and determining the position of the virtual component through some basic mathematical operations. This method is simple, but it ignores the contextual information of the action. For example, when the user performs a series of continuous actions, relying solely on relative position information may not be able to fully capture the continuity of the action and the user's intention, which may cause the position adjustment of the virtual component to be not accurate or natural enough, especially in complex virtual scenes. In addition, relying solely on position data may also fail to capture subtle changes in the user's movements. For example, slight changes in the user's gestures may not be accurately captured, thereby affecting the fineness of the position adjustment of the virtual component, and thus affecting the user's immersive experience.

[0030] In response to the above technical problems, the technical concept of this application is to collect the action video of the VR experiencer in the real space through the VR head-mounted device, collect the time queue of the position data in the real space through the position sensor, and obtain the initial position data of the target object in the real space, and use AI-based video data processing and analysis technology to perform key frame sampling and semantic feature association extraction on the action video, and perform time-series encoding on the position data, so as to intelligently generate the target object position change decoding value based on the query matching between each action-related semantic feature and the position time-series feature, and obtain the interactive feedback result based on the difference between the decoded value and the initial position data. In this way, the relative position of the target object can be adaptively adjusted to better understand the user's action intention and contextual information, capture the user's coherent actions and subtle action changes, and thus more finely and intelligently adjust the position of the target object to provide a more flexible interactive experience.

[0031] Figure 1 : is a system block diagram of a VR interactive system for large space scenes according to an embodiment of the present application. Figure 1 As shown, in the VR interaction system 100 for large space scenes, it includes: an interactive action video position data acquisition module 110, which is used to obtain the action video of the VR experiencer in the real space collected by the VR head-mounted device, and collect the time queue of the position data of the VR experiencer in the real space through the position sensor; an initial position data acquisition module 120, which is used to obtain the initial position data of the target object interacting with the VR experiencer in the real space; a position change value determination module 130, which is used to determine the target object position change decoding value based on the action video and the time queue of the position data; an interactive feedback result generation module 140, which is used to subtract the target object position change decoding value from the initial position data to obtain the relative position change data of the target object as the interactive feedback result.

[0032] In the embodiment of the present application, the interactive action video position data acquisition module 110 and the initial position data acquisition module 120 are respectively used to obtain the action video of the VR experiencer in the real space collected by the VR head-mounted device, and collect the time queue of the position data of the VR experiencer in the real space through the position sensor, and to obtain the initial position data of the target object interacting with the VR experiencer in the real space. It should be understood that the action video of the VR experiencer in the real space records a series of actions of the VR experiencer in the real space, including gestures, body postures, movement methods, etc. These action sequences help to understand the behavior patterns of the VR experiencer and can capture the subtle changes of the VR experiencer. The capture of these details is conducive to the realization of accurate virtual interaction. It also contains information on the interaction between the user and the objects in the real space, and this interaction information can be used to adjust the position and action of the virtual character. The time queue information of the position data of the VR experiencer in the real space reflects the movement trajectory of the user in the real space. By recording the position changes of the user at different time points, the relative movement of the user in the space can be calculated, so as to accurately adjust the position of the virtual object. In general, the combination of the VR experiencer's action video and position data in the real space can provide a comprehensive view of the user's behavior in the real space. By combining the details of the action with the user's position information, the action and position of the virtual character can be adjusted more accurately to ensure that it is consistent with the user's real behavior. The initial position data of the target object interacting with the VR experiencer in the real space provides a reference point, which enables subsequent interactions and position changes to be compared. By understanding the initial position of the target object, the system can more accurately calculate the relative position change between the user and the object, thereby providing interactive feedback.

[0033] In the embodiment of the present application, the position change value determination module 130 is used to determine the target object position change decoding value based on the time sequence of the action video and the position data. Specifically, Figure 2 This is a block diagram of a position change value determination module in a VR interaction system for a large space scene according to an embodiment of the present application. Figure 3 Schematic diagram of data flow of a position change value determination module in a VR interactive system for large space scenes according to an embodiment of the present application. Figure 2 and Figure 3As shown, the position change value determination module 130 includes: an action video key frame sampling unit 131, which is used to perform key frame sampling on the action video to obtain a time queue of action key frames; an action semantic association unit 132, which is used to perform action semantic feature extraction and temporal context encoding on the time queue of the action key frames to obtain a time series of context action associated semantic expression features; a position temporal encoding unit 133, which is used to perform position temporal encoding on the time queue of the position data to obtain position temporal features; a position change query matching optimization representation generation unit 134, which is used to perform local neighborhood optimized query matching on the time series of the context action associated semantic expression features and the position temporal features to obtain a position change query matching optimization representation; a position change decoding unit 135, which is used to obtain the target object position change decoding value based on the position change query matching optimization representation.

[0034] In the embodiment of the present application, the action video key frame sampling unit 131 is used to perform key frame sampling on the action video to obtain a time queue of action key frames. Accordingly, considering that the action video contains a large number of frames, which contain information related to the action, but also correspondingly contains background information, based on this, in order to select representative frames from a series of continuous frames, especially frames containing action changes, so as to better understand the user's action behavior, in the technical solution of the present application, the action video is key frame sampled to obtain a time queue of action key frames.

[0035] In the embodiment of the present application, the action semantic association unit 132 is used to extract action semantic features and perform temporal context encoding on the time queue of the action key frame to obtain a time sequence of context action associated semantic expression features. Specifically, Figure 4 FIG. 1 is a block diagram of an action semantic association unit in a VR interactive system for large space scenes according to an embodiment of the present application. Figure 4 As shown, the action semantic association unit 132 includes: an action semantic feature extraction subunit 1321, which is used to pass the time queue of the action key frames through an action semantic feature extractor based on void convolution coding to obtain a time series of action latent semantic feature vectors; an action semantic feature context encoding subunit 1322, which is used to pass the time series of the action latent semantic feature vectors through an action timing context encoder based on a converter structure to obtain a time series of context action association semantic expression feature vectors as the time series of context action association semantic expression features.

[0036] It should be understood that each action key frame in the time queue of the action key frames contains implicit features about the user's posture and action information at a specific point in time. Considering that the dilated convolutional coding can expand the receptive field without increasing the amount of computation by introducing "holes" (i.e., 0 values) in ordinary convolution kernels, this means that the dilated convolutional neural network can capture a wider range of contextual information, thereby better understanding the semantics of user actions. Therefore, in the technical solution of the present application, the time queue of the action key frames is passed through an action semantic feature extractor based on dilated convolutional coding to capture and mine the action semantic information implicit in each key frame, thereby obtaining a time series of action implicit semantic feature vectors. Then, considering that there are different semantic associations and context influences between different time spans in the time series of the action implicit semantic feature vector, the converter structure can effectively capture and model the contextual relationship between the input sequence data through the self-attention mechanism. Based on this, in order to better capture the long-distance dependencies and contextual information in the action sequence, so as to better understand the semantic coherence between the contexts in which the actions occur, in the technical solution of the present application, the time series of the action implicit semantic feature vector is passed through an action temporal context encoder based on the converter structure to obtain a time series of contextual action associated semantic expression feature vectors.

[0037] In the embodiment of the present application, the position time series encoding unit 133 is used to perform position time series encoding on the time series of the position data to obtain the position time series feature. Specifically, in the embodiment of the present application, the position time series encoding unit is used to: arrange the time series of the position data into the experiencer position time series vector and then input it into the position time series encoder to obtain the position time series feature vector as the position time series feature. It should be understood that the position data changes with time as the experiencer interacts with the target object (interactive object). Therefore, in order to retain the time sequence information of the experiencer's movement trajectory in the real space, so as to better understand the experiencer's behavior pattern, in the technical solution of the present application, the time series of the position data is arranged into the experiencer position time series vector and then input into the position time series encoder to obtain the position time series feature vector. In particular, in a specific implementation method of the embodiment of the present application, the position time series encoder is a convolutional neural network model including a one-dimensional convolutional layer and a fully connected layer, and the convolutional neural network model including a one-dimensional convolutional layer and a fully connected layer is used to perform time series encoding processing on the arranged experiencer position time series vector, thereby obtaining the position time series feature vector.

[0038] In an embodiment of the present application, the position change query matching optimization representation generation unit 134 is used to perform query matching of the time series of the context action associated semantic expression feature and the position timing feature in a local neighborhood optimization to obtain the position change query matching optimization representation. Accordingly, considering that the time series of the context action associated semantic expression feature vector expresses the semantic information of the experiencer when performing a series of actions. The position timing feature vector expresses the position timing information of the experiencer moving in the real space. There is a certain correlation between the semantic information of the action and the temporal change of the position. For example, when the user reaches out to grab the target object (interactive object), the position data of the hand will show a trend of gradually approaching the object. Therefore, in order to identify and query which action features are most relevant to the current position change, so as to more comprehensively understand the relationship between the user's action and position change, in the technical solution of the present application, the time series of the context action associated semantic expression feature vector and the position timing feature vector are subjected to query matching of local neighborhood optimization to obtain the position change query matching action implicit optimization representation vector as the position change query matching optimization representation.

[0039] Specifically, first, the position time series feature vector is processed using a value embedding coding matrix to map the complex time series features to a space that is more suitable for attention analysis and prediction, and a position time series value feature vector is obtained. Then, the time series of the context action associated semantic expression feature vector is subjected to one-dimensional convolutional coding to capture and extract the local semantic neighborhood information in the action sequence, and a sequence of context action associated local neighborhood semantic association feature vectors is obtained. In particular, the size of the one-dimensional convolution kernel is set to an integer multiple of the context action associated semantic expression feature vector to better reflect the semantic correlation between these local neighborhoods. Afterwards, the time series of the context action associated semantic expression feature vector and the sequence of the context action associated local neighborhood semantic association feature vector are used as the sequence of key vectors, the position time series value feature vector is used as the value vector, and the position time series feature vector is used as the query vector, which are respectively input into the first and second multi-head attention query modules based on the converter structure to obtain the sequence of context action associated semantic granularity query matching implicit vectors and the sequence of context action associated semantic neighborhood granularity query matching implicit vectors. It is understandable that the multi-head attention query module can focus on different aspects of the features, thereby capturing the multi-level and complex relationships between features, and thus enhancing the ability of feature representation. Specifically, the first multi-head attention query module focuses on the characteristics of the key vector itself, which means that it mainly focuses on the feature details at each time point, so that it can have a deeper understanding of the specific information at each time point. The second multi-head attention query module focuses on the characteristics of the key vector and its surrounding neighborhood, which means that it not only considers the features at a single time point, but also pays attention to the information near the time point, so that it can capture the relationship between the local and its context. Then, the mean vector of the sequence of semantic granularity query matching implicit vectors and the sequence of semantic neighborhood granularity query matching implicit vectors is calculated to aggregate the features of different semantic granularities to obtain the context action associated semantic granularity query matching implicit feature vector and the context action associated semantic neighborhood granularity query matching implicit feature vector. Finally, the features of the two different granularities are fused to obtain the position change query matching action implicit optimization representation vector, so as to comprehensively express and enhance the action semantic features related to position change.

[0040] Specifically, Figure 5 The block diagram of the position change query matching optimization representation generation unit in the VR interactive system for large space scenes according to the embodiment of the present application. Figure 5As shown, the position change query matching optimization representation generation unit 134 includes: a position time series feature value embedding coding subunit 1341, which is used to process the position time series feature vector using a value embedding coding matrix to obtain a position time series value feature vector; a context action associated semantic expression feature one-dimensional convolution coding subunit 1342, which is used to perform one-dimensional convolution coding on the time series of the context action associated semantic expression feature vector to obtain a sequence of context action associated local neighborhood semantic association feature vectors; an action semantic position query fusion matching subunit 1343, which is used to use the time series of the context action associated semantic expression feature vector and the sequence of the context action associated local neighborhood semantic association feature vector as a sequence of key vectors, and use the position time series value feature vector as a value vector and the position time series feature vector as a query vector, and perform attention-based query fusion matching on the sequence of key vectors, the value vector and the query vector to obtain a position change query matching action implicit optimization representation vector as the position change query matching optimization representation.

[0041] More specifically, Figure 6 FIG. 1 is a block diagram of an action semantic position query fusion matching subunit in a VR interactive system for large space scenes according to an embodiment of the present application. Figure 6As shown, the action semantic position query fusion matching subunit 1343 includes: a context action associated semantic granularity encoding secondary subunit 13431, which is used to use the position time series feature vector as the query vector, the position time series value feature vector as the value vector and the time series of the context action associated semantic expression feature vector as the sequence of key vectors, and input the sequence of the query vector, the value vector and the key vector into a first multi-head attention query module based on a converter structure to obtain a sequence of context action associated semantic granularity query matching implicit vectors; a context action associated semantic neighborhood granularity query matching secondary subunit 13432, which is used to use the position time series feature vector as the query vector, the position time series value feature vector as the value vector and the sequence of the context action associated local neighborhood semantic association feature vector as the sequence of key vectors, and input the query vector, the value vector and the key vector into a first multi-head attention query module based on a converter structure to obtain a sequence of context action associated semantic granularity query matching implicit vectors. The sequence is input into the second multi-head attention query module based on the converter structure to obtain a sequence of context-action associated semantic neighborhood granularity query matching implicit vectors; the context-action associated semantic granularity semantic neighborhood granularity vector mean calculation secondary sub-unit 13433 is used to calculate the sequence of context-action associated semantic granularity query matching implicit vectors and the positional mean vector of the sequence of context-action associated semantic granularity query matching implicit vectors to obtain the context-action associated semantic granularity query matching implicit feature vector and the context-action associated semantic neighborhood granularity query matching implicit feature vector; the context-action associated semantic granularity-semantic neighborhood granularity fusion secondary sub-unit 13434 is used to fuse the context-action associated semantic granularity query matching implicit feature vector and the context-action associated semantic neighborhood granularity query matching implicit feature vector to obtain the position change query matching action implicit optimization representation vector.

[0042] More specifically, in an embodiment of the present application, the context-action association semantic granularity encoding secondary sub-unit is used to: calculate the product between the position-time series feature vector and the transposed vector of the context-action association semantic expression feature vector, divide the obtained position-context action semantic implicit association matrix by the square root of the length of the context-action association semantic expression feature vector by position to obtain a position-context action semantic implicit association modulation matrix; input the position-context action semantic implicit association modulation matrix into a softmax function to obtain a position-context action semantic implicit association activation matrix; multiply the position-context action semantic implicit association activation matrix by the position-time series value feature vector to obtain the context-action association semantic granularity query matching implicit vector.

[0043] More specifically, in an embodiment of the present application, the context-action associated semantic granularity-semantic neighborhood granularity fusion secondary subunit is used to: respectively calculate the positional subtraction, positional dot product and positional addition between the context-action associated semantic granularity query matching implicit feature vector and the context-action associated semantic neighborhood granularity query matching implicit feature vector to obtain the context-action associated semantic granularity-semantic neighborhood granularity difference vector, the context-action associated semantic granularity-semantic neighborhood granularity dot product vector and the context-action associated semantic granularity-semantic neighborhood granularity sum vector; The semantic granularity-semantic neighborhood granularity difference vector, the context action associated semantic granularity-semantic neighborhood granularity dot product vector and the context action associated semantic granularity-semantic neighborhood granularity sum vector are cascaded to obtain a context action associated semantic granularity-semantic neighborhood granularity cascade vector; after performing one-dimensional convolution encoding on the context action associated semantic granularity-semantic neighborhood granularity cascade vector, the obtained context action associated semantic granularity-semantic neighborhood granularity convolution encoding vector is subjected to local window-based maximum pooling processing to obtain the implicit optimized representation vector of the position change query matching action.

[0044] In an embodiment of the present application, the position change query matching optimization representation generation unit is used to perform query matching processing of local neighborhood optimization on the time series of the context action associated semantic expression feature vector and the position time series feature vector according to the following formula; wherein the formula is: ;in, is the position time series feature vector, is the value embedding encoding matrix, is the value bias vector, is the position time series value feature vector, represents the time series of the semantic expression feature vector associated with the context action, is the first, second, ..., first in the time series of the context action associated semantic expression feature vector ,..., context action associated semantic expression feature vector, is a one-dimensional convolutional code, is an integer multiple of the length of the context action associated semantic expression feature vector, is a sequence of semantic association feature vectors of the context action associated local neighborhood, is the first, second, ..., first in the sequence of the context action associated local neighborhood semantic association feature vectors. ,..., context action associated local neighborhood semantic association feature vector, and They are and The transposed vector of and are the lengths of the context action associated semantic expression feature vector and the context action associated local neighborhood semantic association feature vector, respectively, yes function, is the first in the sequence of the context action associated semantic granular query matching implicit vector context action associated semantic granularity query matching latent vector, is the number of feature vectors in the sequence of implicit vectors matched by the context-action-associated semantic granularity query, is the first in the sequence of the context action associated semantic neighborhood granularity query matching implicit vector context action associated semantic neighborhood granularity query matching implicit vector, is the number of feature vectors in the sequence of matching implicit vectors of the context-action associated semantic neighborhood granularity query, is the context action associated semantic granularity query matching implicit feature vector, is the context action associated semantic neighborhood granularity query matching implicit feature vector, , and They are positional subtraction, positional dot multiplication, and positional addition. For cascade processing, is a one-dimensional convolutional coding operation, is the maximum pooling operation, It is the implicit optimization representation vector of the position change query matching action.

[0045] In the embodiment of the present application, the position change decoding unit 135 is used to obtain the target object position change decoding value based on the position change query matching optimization representation. Specifically, in the embodiment of the present application, the position change decoding unit is used to: pass the implicit optimization representation vector of the position change query matching action through the target object position changer based on the decoder to obtain the target object position change decoding value. It should be understood that the position change can be converted into the mobile position data of the target object (interactive object), that is, when the position of the experiencer changes, the position data of the target object will also be updated accordingly. And the action can affect the offset of the relative position between the interactive object and the VR experiencer. That is, the action of the experiencer will affect the change of the interactive object relative to the position of the experiencer, that is, the action of the experiencer will cause the position of the object to shift. Based on this, in order to adjust the position of the interactive object in the real space according to the position change and action of the experiencer, so as to provide a more natural and realistic interactive experience, in the technical solution of the present application, the time series of the context action associated semantic expression feature vector and the position time series feature vector are used to perform query matching optimization to obtain the position change query matching optimization representation for decoding processing, so as to intelligently generate the target object position change decoding value.

[0046] In the embodiment of the present application, the interactive feedback result generating module 140 is used to subtract the decoded value of the target object position change from the initial position data to obtain the relative position change data of the target object as the interactive feedback result. In this way, the relative position of the target object can be adaptively adjusted to better understand the user's action intention and context information, capture the user's coherent actions and subtle action changes, and thus more finely and intelligently adjust the position of the target object to provide a more flexible interactive experience.

[0047] In particular, the present application takes into account the time series of the context-action associated semantic expression feature vector and the position time series feature vector, which respectively represent the full-time context association of the inter-frame semantic sharing degree weighted enhanced image semantic features of the action key frames, and the time series encoding characteristics of the position data. When performing local neighborhood optimized query matching on them, the difference in the dimensionality of multimodal data feature extraction in the time series direction will also make the time-domain fusion representation of the implicitly optimized representation vector of the position change query matching action have a complex spatial structure based on different time series local neighborhood query attention weight distribution structures, thereby expecting to improve its decoding regression convergence and generalization effects under complex spatial structures.

[0048] Preferably, in an example of the present application, the position change query matching action implicit optimization representation vector is passed through a decoder-based target object position changer to obtain a target object position change decoding value, including: calculating the sum of the absolute values ​​of the eigenvalues ​​of the position change query matching action implicit optimization representation vector to obtain a first position change query matching action implicit optimization spatial structure value, and calculating the square root of the sum of the squares of the eigenvalues ​​of the position change query matching action implicit optimization representation vector to obtain a second position change query matching action implicit optimization spatial structure value; multiplying each eigenvalue of the position change query matching action implicit optimization representation vector with the first position change query matching action implicit optimization spatial structure value and the second position change query matching action implicit optimization spatial structure value to obtain a first position change query matching action implicit optimization structure reference value and a second position change query matching action implicit optimization structure reference value corresponding to each eigenvalue; multiplying each eigenvalue of the position change query matching action implicit optimization representation vector with the length of the position change query matching action implicit optimization representation vector and the square root of the length to obtain a first position change query matching action implicit optimization structure reference value and a second position change query matching action implicit optimization structure reference value corresponding to each eigenvalue. The first position change query matching action implicit optimization scale transformation value and the second position change query matching action implicit optimization scale transformation value corresponding to the feature values ​​are calculated; the first position change query matching action implicit optimization structure reference value is divided by the difference between the first position change query matching action implicit optimization space structure value and the first position change query matching action implicit optimization scale transformation value to obtain the first position change query matching action implicit optimization transformation adjustment value; the second position change query matching action implicit optimization structure reference value is divided by the difference between the second position change query matching action implicit optimization space structure value and the second position change query matching action implicit optimization scale transformation value to obtain the second position change query matching action implicit optimization transformation adjustment value; the weighted sum of the first position change query matching action implicit optimization transformation adjustment value and the second position change query matching action implicit optimization transformation adjustment value is calculated to obtain each feature value of the optimized position change query matching action implicit optimization representation vector; the optimized position change query matching action implicit optimization representation vector is passed through a decoder-based target object position changer to obtain a target object position change decoding value.

[0049] Here, the optimized representation of the implicit optimized representation vector of the location change query matching action is: ;in, represents the implicit optimization representation vector of the position change query matching action, Indicates that the position change query matching action implicitly optimizes the representation vector The eigenvalues ​​at the positions, Indicates that the position change query matching action implicitly optimizes the length of the representation vector, Indicates that the first position change query matching action implicitly optimizes the spatial structure value, Indicates that the second position change query matching action implicitly optimizes the spatial structure value, Indicates the first position change query matching action implicitly optimizes the transformation adjustment vector The first position change query matching action implicitly optimizes the transformation adjustment value, Indicates the second position change query matching action implicitly optimizes the transformation adjustment vector The second position change query matching action implicitly optimizes the transformation adjustment value of the position, represents the first position change query matching action implicit optimization transformation adjustment vector, represents the implicit optimization transformation adjustment vector of the second position change query matching action, Indicates adding by position point. It means point multiplication by position. represents the weighted hyperparameter, The optimized position change query matching action implicitly optimizes the representation vector.

[0050] That is, in the preferred example, for the spatial structure information of the feature set of the implicit optimization representation vector of the position change query matching action in the high-dimensional space, the scale-based frame transformation of each feature value of the implicit optimization representation vector of the position change query matching action is performed by using the class norm space structured representation of the implicit optimization representation vector of the position change query matching action as a reference window, and the spatial structure-based frame attention weight adjustment of each feature value of the implicit optimization representation vector of the position change query matching action is realized to ensure the spatial transformation (translation, scaling and rotation) invariance of the implicit optimization representation vector of the position change query matching action under the interaction of the feature space, thereby improving the convergence and generalization effect of the decoding regression of the feature set of the implicit optimization representation vector of the position change query matching action under the complex spatial structure representation, and improving the accuracy of the target object position change decoding value obtained by passing it through the decoder-based target object position changer. In this way, the user's action intention and context information can be better understood, the user's coherent actions and subtle action changes can be captured, and the position of the target object can be adjusted more finely and intelligently to provide a more flexible interactive experience.

[0051] In summary, the VR interactive system 100 for large space scenes based on the embodiment of the present application is explained, which uses AI-based video data processing and analysis technology to perform key frame sampling and semantic feature association extraction on the action video of the VR experiencer in the real space, and performs time series encoding on the time queue of the position data in the real space, so as to intelligently generate the target object position change decoding value according to the query matching between each action-related semantic feature and the position time series feature, and obtain the interactive feedback result based on the difference between the decoding value and the initial position data of the target object in the real space. In this way, the relative position of the target object can be adaptively adjusted to better understand the user's action intention and contextual information, capture the user's coherent actions and subtle action changes, and thus more finely and intelligently adjust the position of the target object to provide a more flexible interactive experience.

[0052] As described above, the VR interaction system 100 for large space scenes according to the embodiment of the present application can be implemented in various terminal devices, such as a server for VR interaction for large space scenes. In one example, the VR interaction system 100 for large space scenes according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the VR interaction system 100 for large space scenes can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the VR interaction system 100 for large space scenes can also be one of the many hardware modules of the terminal device.

[0053] Alternatively, in another example, the VR interaction system 100 for large space scenes and the terminal device may also be separate devices, and the VR interaction system 100 for large space scenes may be connected to the terminal device via a wired and / or wireless network, and transmit interaction information in accordance with an agreed data format.

[0054] Figure 7 FIG. 1 is a flow chart of a VR interaction method for a large space scene according to an embodiment of the present application. Figure 7As shown, in the VR interaction method for large space scenes, it includes: S110, obtaining an action video of a VR experiencer in a real space collected by a VR head-mounted device, and collecting a time queue of the position data of the VR experiencer in the real space through a position sensor; S120, obtaining the initial position data of the target object interacting with the VR experiencer in the real space; S130, determining a target object position change decoding value based on the action video and the time queue of the position data; S140, subtracting the target object position change decoding value from the initial position data to obtain the relative position change data of the target object It is an interactive feedback result; wherein, step S130 includes: performing key frame sampling on the action video to obtain a time queue of action key frames; performing action semantic feature extraction and temporal context encoding on the time queue of action key frames to obtain a time series of context action associated semantic expression features; performing position temporal encoding on the time queue of position data to obtain position temporal features; performing local neighborhood optimized query matching on the time series of context action associated semantic expression features and the position temporal features to obtain a position change query matching optimization representation; based on the position change query matching optimization representation, obtaining the target object position change decoding value.

[0055] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned VR interaction method for large space scenes have been referred to above. Figures 1 to 6 It has been introduced in detail in the description of the VR interaction system for large space scenes, and therefore, its repeated description will be omitted.

[0056] In summary, the VR interaction method for large space scenes based on the embodiment of the present application is explained, which uses AI-based video data processing and analysis technology to perform key frame sampling and semantic feature association extraction on the action video of the VR experiencer in the real space, and performs time series encoding on the time queue of the position data in the real space, so as to intelligently generate the target object position change decoding value based on the query matching between each action-related semantic feature and the position timing feature, and obtain the interactive feedback result based on the difference between the decoding value and the initial position data of the target object in the real space. In this way, the relative position of the target object can be adaptively adjusted to better understand the user's action intention and contextual information, capture the user's coherent actions and subtle action changes, and adjust the position of the target object more finely and intelligently to provide a more flexible interactive experience.

[0057] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details of the above application are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present application to the use of the above specific details to be implemented.

Claims

1. A VR interactive system for large space scenes, characterized in that: include: An interactive action video position data acquisition module is used to acquire an action video of a VR experiencer in a real space collected by a VR head mounted device, and to collect a time queue of position data of the VR experiencer in the real space through a position sensor; An initial position data acquisition module, used to acquire initial position data of a target object interacting with the VR experiencer in the real space; A position change value determination module is used to determine the target object position change decoding value based on the time queue of the action video and the position data, wherein the position change value determination module includes: an action video key frame sampling unit, used to perform key frame sampling on the action video to obtain a time queue of action key frames; an action semantic association unit, used to perform action semantic feature extraction and temporal context encoding on the time queue of the action key frames to obtain a time series of context action associated semantic expression features; a position timing encoding unit, used to perform position timing encoding on the time queue of the position data to obtain position timing features; a position change query matching optimization representation generation unit, used to perform local neighborhood optimized query matching on the time series of the context action associated semantic expression features and the position timing features to obtain a position change query matching optimization representation; a position change decoding unit, used to obtain the target object position change decoding value based on the position change query matching optimization representation; an interactive feedback result generation module, used to subtract the target object position change decoding value from the initial position data to obtain the relative position change data of the target object as the interactive feedback result.

2. The VR interactive system for large space scenes according to claim 1, characterized in that: The action semantic association unit includes: an action semantic feature extraction subunit, which is used to pass the time queue of the action key frames through an action semantic feature extractor based on void convolution coding to obtain a time sequence of action latent semantic feature vectors; an action semantic feature context encoding subunit, which is used to pass the time sequence of the action latent semantic feature vectors through an action timing context encoder based on a converter structure to obtain a time sequence of context action association semantic expression feature vectors as the time sequence of context action association semantic expression features.

3. The VR interactive system for large space scenes according to claim 2 is characterized in that: The position time series encoding unit is used to: arrange the time queue of the position data into an experiencer's position time series vector and then input it into a position time series encoder to obtain a position time series feature vector as the position time series feature.

4. The VR interactive system for large space scenes according to claim 3 is characterized in that: The position change query matching optimization representation generation unit includes: a position time series feature value embedding coding subunit, which is used to process the position time series feature vector using a value embedding coding matrix to obtain a position time series value feature vector; a context action associated semantic expression feature one-dimensional convolution coding subunit, which is used to perform one-dimensional convolution coding on the time series of the context action associated semantic expression feature vector to obtain a sequence of context action associated local neighborhood semantic association feature vectors; an action semantic position query fusion matching subunit, which is used to use the time series of the context action associated semantic expression feature vector and the sequence of the context action associated local neighborhood semantic association feature vector as a sequence of key vectors, and use the position time series value feature vector as a value vector and the position time series feature vector as a query vector, and perform attention-based query fusion matching on the sequence of key vectors, the value vector and the query vector to obtain a position change query matching action implicit optimization representation vector as the position change query matching optimization representation.

5. The VR interactive system for large space scenes according to claim 4, characterized in that: The scale of the one-dimensional convolutional coding is equal to an integer multiple of the length of the context action associated semantic expression feature vector.

6. The VR interactive system for large space scenes according to claim 5, characterized in that: The action semantic position query fusion matching subunit includes: a context action associated semantic granularity encoding secondary subunit, which is used to use the position time series feature vector as the query vector, the position time series value feature vector as the value vector and the time series of the context action associated semantic expression feature vector as the sequence of key vectors, and input the sequence of the query vector, the value vector and the key vector into a first multi-head attention query module based on a converter structure to obtain a sequence of context action associated semantic granularity query matching implicit vectors; a context action associated semantic neighborhood granularity query matching secondary subunit, which is used to use the position time series feature vector as the query vector, the position time series value feature vector as the value vector and the sequence of the context action associated local neighborhood semantic association feature vector as the sequence of key vectors, and input the sequence of the query vector, the value vector and the key vector into a first multi-head attention query module based on a converter structure to obtain a sequence of context action associated semantic granularity query matching implicit vectors. A second multi-head attention query module based on a converter structure is input to obtain a sequence of context-action associated semantic neighborhood granularity query matching implicit vectors; a context-action associated semantic granularity semantic neighborhood granularity vector mean calculation secondary subunit is used to calculate the sequence of context-action associated semantic granularity query matching implicit vectors and the positional mean vector of the sequence of context-action associated semantic neighborhood granularity query matching implicit vectors to obtain context-action associated semantic granularity query matching implicit feature vectors and context-action associated semantic neighborhood granularity query matching implicit feature vectors; a context-action associated semantic granularity-semantic neighborhood granularity fusion secondary subunit is used to fuse the context-action associated semantic granularity query matching implicit feature vectors and the context-action associated semantic neighborhood granularity query matching implicit feature vectors to obtain the position change query matching action implicit optimization representation vector.

7. The VR interactive system for large space scenes according to claim 6, characterized in that: The context-action associated semantic granularity encoding secondary sub-unit is used to: calculate the product between the position-time sequence feature vector and the transposed vector of the context-action associated semantic expression feature vector, divide the obtained position-context action semantic implicit association matrix by the square root of the length of the context-action associated semantic expression feature vector by position to obtain a position-context action semantic implicit association modulation matrix; input the position-context action semantic implicit association modulation matrix into a softmax function to obtain a position-context action semantic implicit association activation matrix; multiply the position-context action semantic implicit association activation matrix by the position-time sequence value feature vector to obtain the context-action associated semantic granularity query matching implicit vector.

8. The VR interactive system for large space scenes according to claim 7, characterized in that: The context action associated semantic granularity-semantic neighborhood granularity fusion secondary subunit is used to: respectively calculate the positional subtraction, positional dot product and positional addition between the context action associated semantic granularity query matching implicit feature vector and the context action associated semantic neighborhood granularity query matching implicit feature vector to obtain the context action associated semantic granularity-semantic neighborhood granularity difference vector, the context action associated semantic granularity-semantic neighborhood granularity dot product vector and the context action associated semantic granularity-semantic neighborhood granularity sum vector; and add the context action associated semantic granularity-semantic neighborhood granularity to the context action associated semantic granularity. The semantic neighborhood granularity difference vector, the context action associated semantic granularity-semantic neighborhood granularity dot product vector and the context action associated semantic granularity-semantic neighborhood granularity sum vector are cascaded to obtain a context action associated semantic granularity-semantic neighborhood granularity cascade vector; after performing one-dimensional convolution encoding on the context action associated semantic granularity-semantic neighborhood granularity cascade vector, the obtained context action associated semantic granularity-semantic neighborhood granularity convolution encoding vector is subjected to local window-based maximum pooling processing to obtain the implicit optimized representation vector of the position change query matching action.

9. The VR interactive system for large space scenes according to claim 8, characterized in that: The position change decoding unit is used to: pass the implicit optimization representation vector of the position change query matching action through a decoder-based target object position changer to obtain the target object position change decoding value.

10. A VR interaction method for large space scenes, characterized in that: The method comprises: obtaining a motion video of a VR experiencer in a real space collected by a VR head mounted device, and collecting a time queue of position data of the VR experiencer in the real space through a position sensor; Acquire initial position data of a target object interacting with the VR experiencer in the real space; Based on the time queue of the action video and the position data, a decoding value of the position change of the target object is determined, including: performing key frame sampling on the action video to obtain a time queue of action key frames; performing action semantic feature extraction and temporal context encoding on the time queue of the action key frames to obtain a time series of context action-associated semantic expression features; performing position temporal encoding on the time queue of the position data to obtain position temporal features; performing local neighborhood optimized query matching on the time series of context action-associated semantic expression features and the position temporal features to obtain a position change query matching optimization representation; obtaining the decoding value of the position change of the target object based on the position change query matching optimization representation; and subtracting the decoding value of the target object position change from the initial position data to obtain the relative position change data of the target object as an interactive feedback result.

Images