Immersive virtual reality display system based on big data technology

Through the full-time domain analysis of user gesture actions, users' interaction intentions are identified and digital human behaviors are controlled, and the problems of inaccurate gesture recognition and insufficient understanding of interaction intentions in immersive virtual reality systems are solved, and the immersion of user experience is improved and Satisfaction.

CN118394218BActive Publication Date: 2024-11-19JIANGXI HANSONG CAR ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410587330.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-13
Publication Date
2024-11-19
Estimated Expiration
2044-05-13

AI Technical Summary

Technical Problem

The existing immersive virtual reality display system has problems such as inaccurate gesture recognition and insufficient understanding of interaction intentions, which affects the user's immersion and satisfaction.

Method used

Through intelligent algorithms, the user's gesture action images at multiple time points in a predetermined time period are analyzed in full time joint analysis, and the user's full time domain interaction behavior pattern is obtained, which is used to judge the user's real interaction intention and control the digital human behavior, improve the interactivity and realism of the system.

Benefits of technology

It realizes more accurate user interaction intention recognition and more realistic digital human behavior, improving the immersion and satisfaction of the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118394218B_ABST
    Figure CN118394218B_ABST
Patent Text Reader

Abstract

The present application discloses an immersive virtual reality display system based on big data technology, which performs full-time joint analysis on gesture action images of a user object at multiple predetermined time points within a predetermined time period through an intelligent algorithm, so as to understand the full-time interactive behavior pattern of the user object, rather than just being limited to the gesture information of a single moment, and uses this full-time interactive behavior pattern information to judge the real interactive intention of the user object, thereby controlling the behavior of a digital human in the immersive virtual reality display system based on big data technology, mapping the interactive intention of the user object to the behavior of the digital human, and improving the interactivity and realism of the system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to an immersive virtual reality display system based on big data technology. Background Art

[0002] The immersive virtual reality display system is a system that uses big data technology, high-speed computing and high-definition display to integrate real scenes and digital characters to provide users with an immersive visual experience. Users can interact with digital characters through gestures, voice and other means to achieve communication, teaching, entertainment and other purposes.

[0003] However, existing immersive virtual reality display systems have some problems, such as inaccurate gesture recognition and insufficient understanding of interaction intentions, which affect the user's immersion and satisfaction. Therefore, an optimized immersive virtual reality display system based on big data technology is expected. Summary of the invention

[0004] In order to solve the above technical problems, this application is proposed. The embodiment of the present application provides an immersive virtual reality display system based on big data technology, which uses an intelligent algorithm to perform a full-time joint analysis of the gesture action images of the user object at multiple predetermined time points within a predetermined time period, so as to understand the full-time interactive behavior pattern of the user object, rather than just the gesture information of a single moment, and use this full-time interactive behavior pattern information to judge the real interactive intention of the user object, and then control the behavior of the digital human in the immersive virtual reality display system based on big data technology, so as to map the interactive intention of the user object to the behavior of the digital human, and improve the interactivity and realism of the system.

[0005] According to one aspect of the present application, an immersive virtual reality display system based on big data technology is provided, which includes:

[0006] A gesture action image acquisition module, used to acquire gesture action images of a user object at multiple predetermined time points within a predetermined time period to obtain a set of gesture action images;

[0007] An interest region visualization module, used for performing gesture interest region visualization processing on the set of gesture action images to obtain a set of masked gesture action images;

[0008] A timing fluctuation analysis module, used for performing timing fluctuation analysis on the set of masked gesture action images to obtain a gesture action timing fluctuation feature vector;

[0009] An interaction intention recognition module, used to determine the interaction intention recognition result of the user object based on the gesture action timing fluctuation feature vector;

[0010] The digital human control module is used to control the behavior of the digital human in the immersive virtual reality display system based on big data technology based on the interaction intention recognition result.

[0011] Compared with the prior art, the present application provides an immersive virtual reality display system based on big data technology, which uses an intelligent algorithm to perform a full-time joint analysis of the gesture action images of the user object at multiple predetermined time points within a predetermined time period, so as to understand the full-time interaction behavior pattern of the user object, rather than just being limited to the gesture information of a single moment, and uses this full-time interaction behavior pattern information to judge the real interaction intention of the user object, thereby controlling the behavior of the digital human in the immersive virtual reality display system based on big data technology, mapping the interaction intention of the user object to the behavior of the digital human, and improving the interactivity and realism of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 is a block diagram of an immersive virtual reality display system based on big data technology according to an embodiment of the present application;

[0014] Figure 2 This is a system architecture diagram of an immersive virtual reality display system based on big data technology according to an embodiment of the present application;

[0015] Figure 3 A block diagram of the training phase of an immersive virtual reality display system based on big data technology according to an embodiment of the present application;

[0016] Figure 4 It is a block diagram of a region of interest visualization module in an immersive virtual reality display system based on big data technology according to an embodiment of the present application;

[0017] Figure 5 It is a block diagram of a timing fluctuation analysis module in an immersive virtual reality display system based on big data technology according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0019] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0020] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0021] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] The existing immersive virtual reality display system has some problems, such as inaccurate gesture recognition and insufficient understanding of interaction intentions, which affect the user's immersion and satisfaction. Therefore, an optimized immersive virtual reality display system based on big data technology is expected.

[0024] In the technical solution of the present application, an immersive virtual reality display system based on big data technology is proposed. Figure 1 It is a block diagram of an immersive virtual reality display system based on big data technology according to an embodiment of the present application. Figure 2 FIG. 1 is a system architecture diagram of an immersive virtual reality display system based on big data technology according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to an embodiment of the present application, the immersive virtual reality display system based on big data technology includes: a gesture action image acquisition module 310, which is used to acquire gesture action images of a user object at multiple predetermined time points within a predetermined time period to obtain a set of gesture action images; an interest region visualization module 320, which is used to perform gesture interest region visualization processing on the set of gesture action images to obtain a set of masked gesture action images; a timing fluctuation analysis module 330, which is used to perform timing fluctuation analysis on the set of masked gesture action images to obtain a gesture action timing fluctuation feature vector; an interaction intention recognition module 340, which is used to determine the interaction intention recognition result of the user object based on the gesture action timing fluctuation feature vector; a digital human control module 350, which is used to control the behavior of the digital human in the immersive virtual reality display system based on big data technology based on the interaction intention recognition result.

[0025] In particular, the gesture action image acquisition module 310 is used to acquire gesture action images of the user object at multiple predetermined time points within a predetermined time period to obtain a set of gesture action images. Considering that in the actual application scenarios of the present application, the interactive behavior of the user object is often continuous and has a certain time sequence. By acquiring the gesture action images of the user object at multiple predetermined time points, the timing information of the gesture action can be obtained, that is, the change of the gesture action over time, which is helpful to analyze the evolution process and trend of the user's interactive behavior, so as to describe and capture the user interaction intention contained in the set of gesture action images.

[0026] In particular, the region of interest visualization module 320 is used to perform gesture region of interest visualization processing on the set of gesture action images to obtain a set of masked gesture action images. In particular, in a specific example of the present application, Figure 4 As shown, the region of interest visualization module 320 includes: a region of interest identification unit 321, used to obtain a set of multiple gesture region of interest images by passing each of the gesture action images through a gesture region of interest identifier based on a sliding anchor window; a mask unit 322, used to mask the set of gesture action images based on the set of multiple gesture region of interest images to obtain the set of masked gesture action images.

[0027] Specifically, the region of interest recognition unit 321 is used to pass each of the gesture action images through a gesture region of interest recognizer based on a sliding anchor window to obtain a set of multiple gesture region of interest images. Here, in the process of obtaining the set of gesture action images, each of the gesture action images may contain a large amount of interference information, such as background and areas irrelevant to gesture recognition, due to environmental interference and other reasons. Therefore, in the technical solution of the present application, each of the gesture action images is passed through a gesture region of interest recognizer based on a sliding anchor window to extract the most important gesture part in each of the gesture action images to obtain a set of multiple gesture region of interest images. It is worth mentioning that the gesture region of interest recognizer is Fast R-CNN, Faster R-CNN or RetinaNet. Among them, Fast R-CNN is an earlier target detection model that improves the speed and accuracy of target detection by introducing a region proposal network. Faster R-CNN is an improvement on Fast R-CNN, which introduces a region proposal network (RPN) to achieve end-to-end target detection. RetinaNet is a single-stage target detection model that aims to solve the shortcomings of single-stage detectors in small target detection.

[0028] Specifically, the masking unit 322 is used to perform masking processing on the set of gesture action images based on the set of the multiple gesture interest region images to obtain the set of masked gesture action images. That is, the set of gesture action images is masked based on the set of the multiple gesture interest region images to obtain the set of masked gesture action images. In this way, irrelevant image content in the gesture action image is removed by masking, thereby reducing a large amount of background interference, so that the subsequent model can focus on feature extraction and recognition of the gesture area during image analysis. Among them, masking processing refers to a technique commonly used in image processing and computer vision for identifying and extracting regions or objects of interest in an image. A mask is usually a binary image of the same size as the original image, in which the value of each pixel is usually 0 or 1, indicating whether the pixel belongs to the region of interest. By creating a mask, a specific region or object in the original image can be identified, so that the system can focus on processing the region of interest and ignore other regions.

[0029] It is worth mentioning that in other specific examples of the present application, the set of gesture action images can also be processed by other methods to visualize the gesture region of interest to obtain a set of masked gesture action images, for example: input the set of gesture action images; use the target detection algorithm or the hand detection algorithm to locate the hand area in the image; use the key point detection algorithm to detect the key points of the hand, so as to determine the position and posture of the hand; according to the detected hand position information, the image is cropped or masked to extract the hand area; a mask is generated according to the position information of the hand area, and the part outside the hand area is set as the background; the generated mask is applied to the original image, only the hand area is retained, and the other parts are set as the background or the hand area is highlighted with a blur effect; the processed image is saved as a new image set, in which each image only contains the gesture area, and the gesture area has been visualized.

[0030] In particular, the timing fluctuation analysis module 330 is used to perform timing fluctuation analysis on the set of masked gesture action images to obtain a gesture action timing fluctuation feature vector. In particular, in a specific example of the present application, Figure 5 The timing fluctuation analysis module 330 includes: a gesture feature extraction unit 331, which is used to extract gesture action features from the set of masked gesture action images using a deep learning network model to obtain a sequence of gesture action semantic feature vectors; a gesture timing feature fluctuation quantization unit 332, which is used to process the sequence of gesture action semantic feature vectors using a fluctuation quantization module to obtain the gesture action timing fluctuation feature vector.

[0031] Specifically, the gesture feature extraction unit 331 is used to extract gesture action features from the set of masked gesture action images using a deep learning network model to obtain a sequence of gesture action semantic feature vectors. That is, in the technical solution of the present application, the set of masked gesture action images is passed through a gesture action feature extractor based on a convolutional neural network model to obtain a sequence of gesture action semantic feature vectors. Here, the convolutional neural network (CNN) is a feedforward neural network, which is characterized by being able to effectively recognize spatial patterns in images. CNN includes a multi-layer neural network, including components such as a convolutional layer, a pooling layer, and an activation function. In the convolutional layer, the network extracts features from the image through a convolution operation; in the pooling layer, the network reduces the dimension of the feature map through a pooling operation; the activation function introduces nonlinearity to increase the network's expression ability. In the technical solution of the present application, the excellent performance of the convolutional neural network in the field of image processing is utilized to learn and extract the implicit features of the gesture action of the image from each masked gesture action image, thereby understanding the gesture meaning of the user object. More specifically, each layer of the gesture action feature extractor based on the convolutional neural network model performs the following operations on the input data in the forward pass of the layer: convolution processing on the input data to obtain a convolution feature map; pooling based on a feature matrix on the convolution feature map to obtain a pooled feature map; and nonlinear activation on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the gesture action feature extractor based on the convolutional neural network model is a sequence of the gesture action semantic feature vectors, and the input of the first layer of the gesture action feature extractor based on the convolutional neural network model is a set of the masked gesture action images.

[0032] Specifically, the gesture timing feature fluctuation quantification unit 332 is used to use the fluctuation quantification module to process the sequence of the gesture action semantic feature vector to obtain the gesture action timing fluctuation feature vector. Considering that when the user object conveys the interaction intention through gesture actions, a continuous gesture action is usually used to express an interaction intention, such as pinching fingers and waving arms, rather than using a static single gesture action to correspond to an interaction intention. That is, the user object may perform a series of gesture actions to represent different instructions or requests, such as swiping left, swiping right, moving up and down, etc. In this way, there is an important correlation between the gesture actions, and this correlation plays an important role in identifying the gesture actions and interaction intentions of the user object. In order to capture and characterize the timing dependency correlation between the gesture actions, in the technical solution of the present application, a fluctuation quantization module is used to process the sequence of the gesture action semantic feature vector to obtain the gesture action timing fluctuation feature vector. The fluctuation quantification module represents the temporal motion characteristics of gestures by measuring and quantifying the degree of fluctuation between the semantic feature vectors of each gesture action and the mean-constrained gesture action representation of the sequence of the semantic feature vectors of the gesture action, thereby capturing important patterns and change information in the sequence of gesture actions, such as turning points and key postures in gesture actions. More specifically, the sequence of the semantic feature vectors of the gesture action is processed using the fluctuation quantification module to obtain the temporal fluctuation feature vector of the gesture action, including: processing the sequence of the semantic feature vectors of the gesture action using the following fluctuation quantification formula to obtain the temporal fluctuation feature vector of the gesture action; wherein the fluctuation quantification formula is:

[0033] in, is the first eigenvalues, The first in the sequence of the semantic feature vector of the gesture action Gesture action semantic feature vectors, The mean feature vector of the sequence representing the semantic feature vector of the gesture action, is the length of the sequence of the gesture action timing fluctuation feature vector

[0034] It is worth mentioning that in other specific examples of the present application, the set of masked gesture action images can also be subjected to time series fluctuation analysis in other ways to obtain a gesture action time series fluctuation feature vector, for example: input the set of masked gesture action images; regard each gesture action image as a time step, and construct a gesture action sequence in chronological order; use a motion tracking algorithm (such as an optical flow method) to analyze the motion information of the hand area in the gesture action sequence to capture the dynamic changes of the gesture; perform feature extraction on the gesture action image of each time step, and use a convolutional neural network to extract image features; combine the features extracted from each time step into a time series feature sequence to form a time series feature representation of the gesture action; perform statistical analysis on the time series feature sequence, such as mean, variance, maximum value, minimum value, etc., to capture the fluctuation characteristics of the gesture action; combine the statistical features into a feature vector as the time series fluctuation feature vector of the gesture action sequence.

[0035] In particular, the interaction intention recognition module 340 is used to determine the interaction intention recognition result of the user object based on the gesture action timing fluctuation feature vector. In particular, in a specific example of the present application, the optimized gesture action timing fluctuation feature vector is passed through a classifier-based interaction intention recognizer to obtain the interaction intention recognition result, and the interaction intention recognition result is used to represent the category label of the interaction action. Specifically, the optimized gesture action timing fluctuation feature vector is fully connected encoded using multiple fully connected layers of the classifier-based interaction intention recognizer to obtain an encoded classification feature vector; and the encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

[0036] A classifier is a machine learning model or algorithm that is used to classify input data into different categories or labels. Classifiers are part of supervised learning and perform classification tasks by learning the mapping relationship from input data to output categories.

[0037] A fully connected layer is a common type of layer in neural networks. In a fully connected layer, each neuron is connected to all neurons in the previous layer, and each connection has a weight. This means that each neuron in the fully connected layer receives inputs from all neurons in the previous layer, and performs a weighted sum of these inputs through the weights, and then passes the result to the next layer.

[0038] The Softmax classification function is a commonly used activation function for multi-classification problems. It converts each element of the input vector into a probability value between 0 and 1, and the sum of these probability values ​​is equal to 1. The Softmax function is often used in the output layer of neural networks, especially for multi-classification problems, because it can map the network output to the probability distribution of each category. During the training process, the output of the Softmax function can be used to calculate the loss function and update the network parameters through the back-propagation algorithm. It is worth noting that the output of the Softmax function does not change the relative size relationship between the elements, but only normalizes them. Therefore, the Softmax function does not change the characteristics of the input vector, but only converts it into a probability distribution form.

[0039] In particular, the digital human control module 350 is used to control the behavior of the digital human in the immersive virtual reality display system based on big data technology based on the interaction intention recognition result. That is, based on the interaction intention recognition result, the behavior of the digital human in the immersive virtual reality display system based on big data technology is controlled. Here, controlling the behavior of the digital human includes but is not limited to action adjustment (such as waving and smiling), voice interaction (such as saying "hello") and posture adjustment (such as facing the user), etc.

[0040] It should be understood that before using the above-mentioned neural network model for inference, it is necessary to train the gesture region of interest identifier based on the sliding anchor window, the gesture action feature extractor based on the convolutional neural network model, the fluctuation quantization module and the classifier-based interaction intention identifier. That is to say, according to the immersive virtual reality display system 300 based on big data technology of the present application, it also includes a training stage 400 for training the gesture region of interest identifier based on the sliding anchor window, the gesture action feature extractor based on the convolutional neural network model, the fluctuation quantization module and the classifier-based interaction intention identifier.

[0041] Figure 3 FIG. 1 is a block diagram of the training phase of an immersive virtual reality display system based on big data technology according to an embodiment of the present application. Figure 3As shown, according to an immersive virtual reality display system 300 based on big data technology according to an embodiment of the present application, it includes: a training stage 400, including: a training data acquisition unit 410, used to acquire training data, the training data including training gesture action images of a user object at multiple predetermined time points within a predetermined time period to obtain a set of training gesture action images, and the true value of the category label of the interactive action; a training interest region recognition unit 420, used to pass each of the training gesture action images through the gesture interest region recognizer based on the sliding anchor window to obtain a set of multiple training gesture interest region images; a training mask unit 430, used to mask the set of training gesture action images based on the set of multiple training gesture interest region images to obtain a set of training masked gesture action images; a training gesture action feature extraction unit 440, used to pass the set of training masked gesture action images through the The gesture action feature extractor based on the convolutional neural network model is used to obtain a sequence of training gesture action semantic feature vectors; a training time series feature fluctuation quantization unit 450 is used to use the fluctuation quantization module to process the sequence of training gesture action semantic feature vectors to obtain training gesture action time series fluctuation feature vectors; an optimization unit 460 is used to cluster and optimize the training gesture action time series fluctuation feature vectors to obtain optimized training gesture action time series fluctuation feature vectors; a classification loss unit 470 is used to pass the optimized training gesture action time series fluctuation feature vectors through the classifier-based interaction intention identifier to obtain a classification loss function value; a training unit 480 is used to train the gesture interest region identifier based on the sliding anchor window, the gesture action feature extractor based on the convolutional neural network model, the fluctuation quantization module and the classifier-based interaction intention identifier with the classification loss function value.

[0042] In the technical solution of the present application, each training gesture action semantic feature vector in the sequence of training gesture action semantic feature vectors respectively represents the image semantic features of the masked gesture action image. However, considering that each masked training gesture action image in the set of masked training gesture action images has obvious source domain representation differences in the image source domain, and there are other interference factors in the masked training gesture action image, for example, the position of the gesture in the masked training gesture action image. Therefore, in the technical solution of the present application, each training gesture action semantic feature vector in the sequence of training gesture action semantic feature vectors has obvious feature distribution inconsistency in the data manifold of the high-dimensional feature space. When the fluctuation quantization module is used to process the sequence of training gesture action semantic feature vectors, its essence is to use the fluctuation quantization module to measure the fluctuation degree of each training gesture action semantic feature vector in the sequence of training gesture action semantic feature vectors relative to the cluster center of the gesture action mean semantic distribution to represent the motion information of the gesture. However, there is an obvious inconsistency in the feature distribution of each training gesture action semantic feature vector in the sequence of the training gesture action semantic feature vector in the data manifold of the high-dimensional feature space, which will cause the training gesture action timing fluctuation feature vector to have a significant local feature distribution discreteness. In this way, when the training gesture action timing fluctuation feature vector is classified as a whole by the classifier-based interactive intention recognizer, the local feature distribution discreteness of the training gesture action timing fluctuation feature vector will lead to the difficulty of convergence to the predetermined class probability when performing class regression by the classifier-based interactive intention recognizer, thereby affecting the training speed of the classifier-based interactive intention recognizer and the accuracy of the final classification result. Based on this, the applicant of the present application performs clustering optimization on the training gesture action timing fluctuation feature vector, that is, first clustering each eigenvalue of the training gesture action timing fluctuation feature vector, such as clustering based on the distance between eigenvalues, and then optimizing based on the in-class and out-of-class representations of the clustered features, expressed as:

[0043] in, are the eigenvalues ​​of the training gesture action timing fluctuation eigenvector, is the number of feature sets corresponding to the training gesture action timing fluctuation feature vector, is the number of clustering features, Represents a cluster feature set. Specifically, by taking the in-class features and out-of-class features of the training gesture action timing fluctuation feature vector as different instance roles to perform a class instance description based on clustering proportion distribution, and introducing the clustering response history based on the in-class and out-of-class dynamic context, a coordinated global perspective is maintained on the in-class distribution and out-of-class distribution of the overall features of the training gesture action timing fluctuation feature vector, so that the optimized feature clustering operation for the training gesture action timing fluctuation feature vector can maintain a consistent response of the in-class and out-of-class features, so that the regression convergence path based on feature clustering remains consistent during the class regression process, and the convergence effect of the training gesture action timing fluctuation feature vector towards the predetermined class probability is improved, so as to improve the training speed of the classifier-based interaction intention identifier and the accuracy of the classification results.

[0044] As described above, the immersive virtual reality display system 300 based on big data technology according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an immersive virtual reality display algorithm based on big data technology. In a possible implementation, the immersive virtual reality display system 300 based on big data technology according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the immersive virtual reality display system 300 based on big data technology can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the immersive virtual reality display system 300 based on big data technology can also be one of the many hardware modules of the wireless terminal.

[0045] Alternatively, in another example, the immersive virtual reality display system 300 based on big data technology and the wireless terminal may also be separate devices, and the immersive virtual reality display system 300 based on big data technology may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0046] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An immersive virtual reality display system based on big data technology, characterized in that: include: A gesture action image acquisition module, used to acquire gesture action images of a user object at multiple predetermined time points within a predetermined time period to obtain a set of gesture action images; An interest region visualization module, used for performing gesture interest region visualization processing on the set of gesture action images to obtain a set of masked gesture action images; A timing fluctuation analysis module, used for performing timing fluctuation analysis on the set of masked gesture action images to obtain a gesture action timing fluctuation feature vector; An interaction intention recognition module, used to determine the interaction intention recognition result of the user object based on the gesture action timing fluctuation feature vector; A digital human control module, used to control the behavior of the digital human in the immersive virtual reality display system based on big data technology based on the interaction intention recognition result; Wherein, the timing fluctuation analysis module includes: A gesture feature extraction unit, configured to extract gesture features from the set of masked gesture action images using a deep learning network model to obtain a sequence of gesture action semantic feature vectors; A gesture timing feature fluctuation quantization unit, used for processing the sequence of the gesture action semantic feature vectors using a fluctuation quantization module to obtain the gesture action timing fluctuation feature vector; Wherein, the gesture timing feature fluctuation quantification unit is used for: The sequence of the gesture action semantic feature vectors is processed using the following fluctuation quantization formula to obtain the gesture action temporal fluctuation feature vector; wherein the fluctuation quantization formula is: ;in, is the first eigenvalues, The first in the sequence of the semantic feature vector of the gesture action Gesture action semantic feature vectors, The mean feature vector of the sequence representing the semantic feature vector of the gesture action, is the length of the sequence of the gesture action timing fluctuation feature vectors.

2. The immersive virtual reality display system based on big data technology according to claim 1 is characterized in that: The region of interest visualization module includes: An area of ​​interest recognition unit, used for passing each of the gesture action images through a gesture area of ​​interest recognizer based on a sliding anchor window to obtain a set of multiple gesture area of ​​interest images; The masking unit is used to perform masking processing on the set of gesture action images based on the set of the plurality of gesture region of interest images to obtain the set of masked gesture action images.

3. The immersive virtual reality display system based on big data technology according to claim 2 is characterized in that: The gesture region of interest identifier is Fast R-CNN, Faster R-CNN or RetinaNet.

4. The immersive virtual reality display system based on big data technology according to claim 3 is characterized in that: The gesture feature extraction unit is used to: The set of masked gesture action images is passed through a gesture action feature extractor based on a convolutional neural network model to obtain a sequence of the gesture action semantic feature vectors.

5. The immersive virtual reality display system based on big data technology according to claim 4 is characterized in that: The interaction intention recognition module is used to: The optimized gesture action timing fluctuation feature vector is passed through a classifier-based interaction intention identifier to obtain the interaction intention recognition result, and the interaction intention recognition result is used to represent the category label of the interaction action.

6. The immersive virtual reality display system based on big data technology according to claim 5 is characterized in that: The interaction intention recognition module includes: a fully connected encoding unit, configured to perform fully connected encoding on the optimized gesture action timing fluctuation feature vector using a plurality of fully connected layers of the classifier-based interaction intention identifier to obtain an encoded classification feature vector; and The classification result generating unit is used to pass the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

7. The immersive virtual reality display system based on big data technology according to claim 6 is characterized in that: The invention also provides a training module for training the gesture region of interest identifier based on the sliding anchor window, the gesture action feature extractor based on the convolutional neural network model, the fluctuation quantization module and the classifier-based interaction intention identifier.

8. The immersive virtual reality display system based on big data technology according to claim 7 is characterized in that: The training module comprises: A training data acquisition unit, used to acquire training data, wherein the training data includes training gesture action images of the user object at multiple predetermined time points within a predetermined time period to obtain a set of training gesture action images, and a true value of a category label of an interactive action; A training region of interest recognition unit, used for passing each of the training gesture action images through the gesture region of interest recognizer based on the sliding anchor window to obtain a set of multiple training gesture region of interest images; A training mask unit, configured to perform mask processing on the set of training gesture action images based on the set of the plurality of training gesture region of interest images to obtain a set of training masked gesture action images; A training gesture action feature extraction unit, used for passing the set of training masked gesture action images through the gesture action feature extractor based on the convolutional neural network model to obtain a sequence of training gesture action semantic feature vectors; A training time series feature fluctuation quantization unit, used for processing the sequence of the training gesture action semantic feature vectors using the fluctuation quantization module to obtain a training gesture action time series fluctuation feature vector; An optimization unit, used for performing cluster optimization on the training gesture action timing fluctuation feature vector to obtain an optimized training gesture action timing fluctuation feature vector; A classification loss unit, used for passing the optimized training gesture action time series fluctuation feature vector through the classifier-based interaction intention identifier to obtain a classification loss function value; A training unit is used to train the gesture region of interest identifier based on the sliding anchor window, the gesture action feature extractor based on the convolutional neural network model, the fluctuation quantization module and the classifier-based interaction intention identifier with the classification loss function value.