An immersive paper shadow performance AR interactive system
By using the optimization methods of gesture recognition network and speech recognition model in the AR interactive system, combining the pseudo-3D convolutional layer and channel attention mechanism, the problem of single interaction mode and complex calculation is solved, and the interactivity of paper shadow performance is improved and Identification accuracy optimizes the utilization of computing resources.
Patent Information
- Application Number
- CN202411031698.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-30
AI Technical Summary
The existing AR interaction system has a single interaction method, lacks natural human-computer interaction, and cannot fully reflect the interactivity and fun of paper shadow performance. The gesture recognition algorithm is too many parameters and complex in the calculation when deployed on mobile devices, and the speech recognition model is easy. It falls into local optimal solution and consumes a lot of computing resources.
The gesture recognition network is used to combine pseudo-3D residual convolution layer and channel attention mechanism to extract spatiotemporal features, design a multi-scale motion attention mechanism, optimize the hyperparameters of the speech recognition model through spider monkey algorithm and simulated annealing algorithm to achieve more efficient interaction and Identify.
It improves the audience's sense of participation and interactivity, improves gesture recognition accuracy and efficiency, avoids the local optimal solution of the speech recognition model, and optimizes the utilization of computing resources.
Smart Images

Figure CN118550416B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of virtual reality technology, and in particular to an immersive paper shadow performance AR interaction system. Background Art
[0002] Traditional shadow puppet shows are usually performed on a fixed stage. The interaction between the audience and the performers is limited, and the audience participation is low. The interactive method of shadow puppet shows based on AR is an emerging interactive method. The existing AR interactive system mainly relies on devices such as touch screens or controllers for interaction. The interactive method is single and lacks natural human-computer interaction. It cannot fully reflect the interactivity and fun of shadow puppet shows. An AR interactive system combining gesture recognition and speech recognition is proposed. However, general gesture recognition algorithms have problems of too many parameters and high computational complexity when deployed on mobile and embedded devices. The hyperparameter search algorithm of general speech recognition models is prone to fall into local optimal solutions and cannot find the global optimal solution, resulting in poor model performance and consumption of a large amount of computing resources. Summary of the invention
[0003] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides an immersive paper shadow performance AR interaction system. In view of the problem that the existing AR interaction system mainly relies on devices such as touch screens or controllers for interaction, has a single interaction mode, lacks natural human-computer interaction mode, and cannot fully reflect the interactivity and fun of paper shadow performances, this scheme proposes an immersive paper shadow performance AR interaction system, establishes a gesture recognition network to identify the audience's gesture state, and enters voice wake-up when the audience is recognized as a fist gesture. The audience interacts with the virtual paper shadow character through voice commands. When the audience is recognized as a stabbing gesture, a paper shadow science video is played, which greatly improves the audience's sense of participation and interactivity and brings innovation to the performance form of paper shadow play. When general gesture recognition algorithms are deployed on mobile and embedded devices, there are too many parameters and high computational complexity. To address this problem, this solution extracts spatiotemporal features from input features through a pseudo 3D residual convolutional layer, and introduces a channel attention mechanism into it, which effectively extracts spatiotemporal feature information while reducing the complexity of the model. A multi-scale motion attention mechanism is designed to construct explicit motion features, and by calculating the feature differences between adjacent frames, the model is helped to better understand the dynamic information of gesture movements and improve recognition accuracy and efficiency. The hyperparameter search algorithm of a general speech recognition model is prone to fall into the local optimal solution and cannot find the global optimal solution, resulting in poor model performance and consuming a lot of computing resources. This solution uses the spider monkey algorithm to perform global and local searches to find the local optimal solution and the global optimal solution, and then uses the probability acceptance mechanism of the simulated annealing algorithm to perform local searches around the global optimal solution to further optimize the quality of the solution, avoid falling into the local optimal solution, and improve the optimization efficiency.
[0004] The present invention provides an immersive paper shadow performance AR interactive system, which includes a paper shadow animation production module, a virtual environment building module, and a user interaction design module;
[0005] The paper shadow animation production module collects paper shadow image materials, processes the collected paper shadow image materials, produces paper shadow animations through skeleton animation technology, and sends the paper shadow animations to the virtual environment construction module;
[0006] The virtual environment building module uses the AR development platform to build a virtual environment, imports the paper shadow play animation as a virtual object into the virtual environment, and sends the virtual environment to the user interaction design module;
[0007] The user interaction design module includes: a gesture recognition module, a voice recognition module, a command parsing module, a video playback module and a paper shadow control module;
[0008] The gesture recognition module establishes a gesture recognition network to recognize the user's gesture state, and when a specific interactive gesture of the user is recognized, enters a corresponding processing flow according to the gesture type;
[0009] The speech recognition module, when recognizing that the user is in a preset first gesture type, develops a speech recognition model, recognizes the user's voice command, and sends the user's voice command to the command parsing module;
[0010] The command parsing module parses the user's voice command into specific operation instructions, and sends the specific operation instructions to the paper shadow control module;
[0011] The paper shadow control module controls the paper shadow in the virtual environment to perform according to specific operation instructions;
[0012] When the video playing module recognizes that the user has a preset second gesture type, the paper shadow science video is played.
[0013] Furthermore, the paper shadow animation production module collects paper shadow image materials, uses Photoshop software to process paper shadow images, extracts paper shadows from paper shadow images, further separates the limbs of paper shadows according to the needs of paper shadow play animation, and after preliminary processing of the paper shadow images, uses skeleton animation to produce shadow animation in Adobe Animate software. Skeleton animation uses the principle of inverse dynamics to first place the posture of the paper shadow character with the mouse, then reversely calculates the motion information and global transformation information of the parent skeleton, and finally generates shadow animation through time series updates.
[0014] Furthermore, the gesture recognition module establishes a gesture recognition network to recognize the user's gesture state, including the following steps:
[0015] Step S1: data collection and preprocessing, collecting gesture video data, preprocessing the gesture video data, and obtaining preprocessed video data;
[0016] Step S2: Data labeling: labeling the video frames in the preprocessed video data, clearly indicating the gesture category in each frame, and dividing it into a training set and a test set;
[0017] Step S3: Establish a gesture recognition network. The gesture recognition network includes: a convolution layer, a pseudo 3D convolution layer, a channel attention layer, a feature fusion layer, and a multi-scale motion attention layer. The steps of constructing the gesture recognition network are as follows:
[0018] Step S31: Input the training set Convolution layer, then pass through the pseudo 3D convolution layer, and then pass through a Convolutional layer, to obtain the output feature map of the pseudo 3D convolutional layer, the pseudo 3D convolutional layer includes two convolutional layers: one A convolutional layer and a Convolutional layers, The convolutional layer is responsible for extracting features in the time dimension, while The convolutional layer is responsible for extracting features in the spatial dimension;
[0019] Step S32: Integrate the channel attention mechanism into the channel attention layer, model the channel relationship of the output feature map of the pseudo 3D convolutional layer, and obtain the channel information weight distribution of the feature. The steps are as follows:
[0020] Step S321: Obtain spatial information, perform spatial average pooling on the output feature map of the pseudo 3D convolutional layer to obtain spatial information. The formula used is as follows:
[0021] ;
[0022] In the formula, Represents spatial information. and Represent the height and width of the feature map respectively, represents the output feature map of the pseudo 3D convolutional layer, It means that the time step is selected from the output feature map of the pseudo 3D convolution layer. , the spatial position is The pixel value of
[0023] Step S322: Compress the number of channels using The convolution kernel convolves the spatial information and compresses the number of channels to the original , get the compressed feature map, the formula used is as follows:
[0024] ;
[0025] In the formula, represents the compressed feature map, express Convolution kernel;
[0026] Step S323: Perform temporal reasoning to reconstruct the compressed feature map into a feature map with the same temporal dimension as the output feature map of the pseudo 3D convolutional layer, and obtain the reconstructed feature map. The convolution kernel processes the reconstructed feature map to obtain the feature map after time inference. The formula used is as follows:
[0027] ;
[0028] In the formula, represents the feature map after temporal reasoning, represents the reconstructed feature map, express Convolution kernel;
[0029] Step S324: Obtain channel excitation matrix, using The convolution kernel and activation function process the feature map after temporal reasoning to obtain the channel excitation matrix. The formula used is as follows:
[0030] ;
[0031] ;
[0032] In the formula, Indicates the use The feature map obtained by the convolution kernel processing the feature map after temporal reasoning, express Convolution kernel, represents the channel excitation matrix, express Activation function;
[0033] Step S325: Calculate the output features of the channel attention layer. The formula used is as follows:
[0034] ;
[0035] In the formula, represents the output features of the channel attention layer, represents feature fusion operation;
[0036] Step S33: fusing the output features of the pseudo 3D convolution layer and the output features of the channel attention layer through the feature fusion layer to obtain the final output features;
[0037] Step S34: Introduce a multi-scale motion attention mechanism. Design a multi-scale motion attention mechanism in the multi-scale motion attention layer to construct explicit motion features. By calculating the feature differences between adjacent frames, pay attention to and extract the temporal feature information of the entire gesture motion process. The steps are as follows:
[0038] Step S341: Feature dimensionality reduction, the final output feature is Two-dimensional convolution is used to reduce the dimension, and we get ,in Indicates the final output feature at time step , and Feature map on ;
[0039] Step S342: Calculate the difference between two adjacent frames of the final output feature map to obtain a difference map. The calculation process is as follows:
[0040] ;
[0041] In the formula, Represents the time step The input feature map is Represents the time step The input feature map is express Convolution operation, represents a difference map;
[0042] Step S343: Process the difference graph and stack it according to the time dimension In order to deal with the position offset problem between the feature maps of two adjacent frames, we use Spatial convolution is used to fuse the position information of the feature maps of two adjacent frames and then spliced into the time dimension. The video data of the frame can only be generated Frame difference map, the first Zero padding of frames to obtain a complete difference map;
[0043] Step S344: multi-scale pooling, input the complete difference map into the multi-scale pooling layer, perform downsampling operation through pooling kernels of different scales to obtain a multi-scale pooling difference map, and The activation function generates the motion attention weight coefficient;
[0044] Step S345: obtaining the output of the multi-scale motion attention layer, multiplying the attention weight coefficient by the final output feature element by element, and obtaining the output feature of the multi-scale motion attention layer;
[0045] Step S4: Model training and evaluation: Use the output features of the multi-scale motion attention layer to train the gesture recognition network to obtain the trained gesture recognition network. Use the test set to evaluate the performance of the trained gesture recognition network, output the probability score of the gesture action category, and select the highest probability score as the prediction result.
[0046] Furthermore, the developing of the speech recognition model to recognize the user's speech command comprises the following steps:
[0047] Step B1: Data collection and preprocessing, collecting audio samples including different intonations, performing audio segmentation, removing silent parts, applying audio enhancement techniques, enhancing audio samples, adopting audio synthesis strategies, creating synthetic audio samples, and labeling and annotating synthetic audio samples;
[0048] Step B2: Dataset division and feature extraction: divide the synthesized audio samples into training set and test set.
[0049] Use Mel spectrum analysis to extract features;
[0050] Step B3: Model construction, establish a speech recognition model. The model starts from the input layer, performs preliminary feature extraction through a one-dimensional convolution layer, introduces a batch normalization layer and a ReLU activation function, and is supplemented by a Dropout layer for regularization. Then it passes through two GRU layers. Each GRU layer is followed by a Dropout layer, a batch normalization layer, and an attention mechanism layer. At the end of the model, two output layers are designed: one uses a sigmoid activation function to predict existence, and the other uses a softmax activation function to distinguish categories, forming the final output of the model;
[0051] Step B4: Model training, using the training set to train the speech recognition model;
[0052] Step B5: Model tuning: Use a hybrid optimization algorithm to optimize the hyperparameters of the speech recognition model to obtain an optimized speech recognition model. The steps are as follows:
[0053] Step B51: Initialize parameters to determine the spider monkey population size, disturbance rate, local and global leader limits, maximum number of groups, and termination conditions;
[0054] Step B52: Evaluate the initial solution, assign an initial position to each spider monkey, and calculate the fitness value of each spider monkey using the fitness function. The formula used is as follows:
[0055] ;
[0056] In the formula, Indicates The fitness value of a spider monkey, Indicates The objective function value of the spider monkey is represents the absolute value function;
[0057] Step B53: Find local and global leaders, determine the local leader of each group according to the fitness value, and determine the global leader from all local leaders;
[0058] Step B54: In the local leader stage, the position of each spider monkey is updated according to the position of the local leader and the randomly selected spider monkey. The formula used is as follows:
[0059] ;
[0060] In the formula, Represents the updated spider monkey In the In the iteration The new location of the Indicates Spider monkeys in The current location of the dimension, Indicates The local leader of the group is The location of the dimension, Indicates that from A randomly selected spider monkey in the group The location of the dimension, Indicates from or A random number chosen uniformly in the range;
[0061] Step B55: In the global leader phase, the position of each spider monkey is updated according to the position of the global leader and the randomly selected spider monkey. The formula used is as follows:
[0062] ;
[0063] In the formula, The global optimal spider monkey is The location of the dimension;
[0064] Step B56: Use simulated annealing algorithm to search for the global optimal solution. The steps are as follows:
[0065] Step B561: Use the simulated annealing algorithm to search around the global optimal solution, generate a scaling factor, and calculate the temperature value. Generate a new solution based on the temperature value, and calculate the fitness value of the new solution to find the optimal fitness value in each iteration. The formula used is as follows:
[0066] ;
[0067] In the formula, represents the average fitness value of spider monkeys around the global optimal solution, represents the sum of the fitness values of all spider monkeys, represents the spider monkey population, represents the fitness value of each solution considered in the simulated annealing process;
[0068] Step B562: Compare the fitness value of the new solution with the current solution. If the new solution is better than the current solution, accept the new solution and increase the number of iterations.
[0069] Step B563: Check whether the number of iterations has reached the maximum number of iterations. If so, update the new solution, update the position of the spider monkey according to the new solution, pay attention to the constraints of the local and global leaders while updating the position, and determine whether the termination condition is met. If so, obtain the output and terminate the algorithm. If the termination condition is not met, go to step B52 to continue execution; if the maximum number of iterations is not reached, go to step B561 to continue execution;
[0070] Step B564: If the new solution is not as good as the current solution, the new solution is accepted or rejected according to the acceptance probability. The acceptance probability is calculated using the following formula:
[0071] ;
[0072] In the formula, represents the probability of acceptance, Indicates the difference between the new solution and the current solution;
[0073] Generate a random number between 0 and 1. If the random number is less than the acceptance probability, accept the new solution, increase the number of iterations, and go to step B563 to continue. Otherwise, reject the new solution, accept the current solution, and continue iterating to execute step B563.
[0074] Step B6: Model evaluation: Use the test set to evaluate the optimized speech recognition model, and use accuracy and F1 score as evaluation indicators to evaluate the model performance.
[0075] The beneficial effects achieved by the present invention using the above scheme are as follows:
[0076] (1) In view of the problem that the existing AR interactive system mainly relies on devices such as touch screens or controllers for interaction, the interaction mode is single, lacks natural human-computer interaction mode, and cannot fully reflect the interactivity and fun of paper shadow performances, this scheme proposes an immersive paper shadow performance AR interactive system, establishes a gesture recognition network to identify the audience's gesture state, and enters voice wake-up when the audience is recognized as a fist gesture. The audience interacts with the virtual paper shadow character through voice commands. When the audience is recognized as a stabbing gesture, a paper shadow science video is played, which greatly improves the audience's sense of participation and interactivity, and brings innovation to the performance form of paper shadow play;
[0077] (2) In view of the problems of too many parameters and high computational complexity when general gesture recognition algorithms are deployed on mobile and embedded devices, this solution extracts spatiotemporal features from the input features through a pseudo 3D residual convolution layer and introduces a channel attention mechanism to effectively extract spatiotemporal feature information while reducing the complexity of the model. A multi-scale motion attention mechanism is designed to construct explicit motion features. By calculating the feature differences between adjacent frames, the model is helped to better understand the dynamic information of gesture motion and improve recognition accuracy and efficiency.
[0078] (3) The hyperparameter search algorithm of general speech recognition models is prone to falling into local optimal solutions and being unable to find the global optimal solution, resulting in poor model performance and consuming a large amount of computing resources. This solution uses the spider monkey algorithm to perform global and local searches to find local and global optimal solutions, and then uses the probability acceptance mechanism of the simulated annealing algorithm to perform local searches around the global optimal solution to further optimize the quality of the solution, avoid falling into local optimal solutions, and improve optimization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 This is a structural framework diagram of an immersive paper shadow performance AR interactive system proposed by the present invention;
[0080] Figure 2 This is a schematic diagram of the structure of the gesture recognition network proposed by the present invention;
[0081] Figure 3 Schematic diagram of the flow of the hybrid optimization algorithm.
[0082] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION
[0083] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0084] Example 1, see Figure 1 , an immersive paper shadow performance AR interactive system provided by the present invention includes a paper shadow animation production module, a virtual environment building module, and a user interaction design module;
[0085] The paper shadow animation production module collects paper shadow image materials, processes the collected paper shadow image materials, produces paper shadow animations through skeleton animation technology, and sends the paper shadow animations to the virtual environment construction module;
[0086] The virtual environment building module uses the AR development platform to build a virtual environment, imports the paper shadow play animation as a virtual object into the virtual environment, and sends the virtual environment to the user interaction design module;
[0087] The user interaction design module includes: a gesture recognition module, a voice recognition module, a command parsing module, a video playback module and a paper shadow control module;
[0088] The gesture recognition module establishes a gesture recognition network to recognize the user's gesture state, and when a specific interactive gesture of the user is recognized, enters a corresponding processing flow according to the gesture type;
[0089] The speech recognition module, when recognizing that the user is in a preset first gesture type, develops a speech recognition model, recognizes the user's voice command, and sends the user's voice command to the command parsing module;
[0090] The command parsing module parses the user's voice command into specific operation instructions, and sends the specific operation instructions to the paper shadow control module;
[0091] The paper shadow control module controls the paper shadow in the virtual environment to perform according to specific operation instructions;
[0092] When the video playing module recognizes that the user has a preset second gesture type, the paper shadow science video is played.
[0093] The preset first gesture type is a fist gesture, and the preset second gesture type is a stabbing gesture.
[0094] By performing the operations, in view of the problem that the existing AR interactive system mainly relies on devices such as touch screens or controllers for interaction, has a single interactive mode, lacks natural human-computer interaction, and cannot fully reflect the interactivity and fun of paper shadow performances, this scheme proposes an immersive paper shadow performance AR interactive system, establishes a gesture recognition network to identify the audience's gesture state, enters voice wake-up when the audience is recognized as a fist gesture, and the audience interacts with the virtual paper shadow character through voice commands. When the audience is recognized as a stabbing gesture, a paper shadow science video is played, which greatly improves the audience's sense of participation and interactivity and brings innovation to the performance form of paper shadow play.
[0095] Embodiment 2. This embodiment is based on the above embodiment. The paper shadow animation production module collects paper shadow image materials, uses Photoshop software to process paper shadow images, extracts paper shadows from paper shadow images, and further separates the limbs of paper shadows according to the needs of paper shadow animation. After preliminary processing of the paper shadow images, in Adobe Animate software, skeleton animation is used to produce shadow animation. Skeleton animation uses the principle of inverse dynamics. First, the posture of the paper shadow character is placed by the mouse, and then the motion information and global transformation information of the upper-level skeleton are reversely calculated. Finally, the shadow animation is generated through time series updates.
[0096] Example 3, see Figure 2 This embodiment is based on the above embodiment, and the gesture recognition module establishes a gesture recognition network to recognize the user's gesture state, including the following steps:
[0097] Step S1: data collection and preprocessing, collecting video data containing a stabbing gesture and a fisting gesture, and preprocessing the video data to obtain preprocessed video data;
[0098] Step S2: Data labeling: labeling the video frames in the preprocessed video data, clearly indicating the gesture category in each frame, and dividing it into a training set and a test set;
[0099] Step S3: Establish a gesture recognition network. The gesture recognition network includes: a convolution layer, a pseudo 3D convolution layer, a channel attention layer, a feature fusion layer, and a multi-scale motion attention layer. The steps of constructing the gesture recognition network are as follows:
[0100] Step S31: Input the training set Convolution layer, then pass through the pseudo 3D convolution layer, and then pass through a Convolutional layer, to obtain the output feature map of the pseudo 3D convolutional layer, the pseudo 3D convolutional layer includes two convolutional layers: one A convolutional layer and a Convolutional layers, The convolutional layer is responsible for extracting features in the time dimension, while The convolutional layer is responsible for extracting features in the spatial dimension;
[0101] Step S32: Integrate the channel attention mechanism into the channel attention layer, model the channel relationship of the output feature map of the pseudo 3D convolutional layer, and obtain the channel information weight distribution of the feature. The steps are as follows:
[0102] Step S321: Obtain spatial information, perform spatial average pooling on the output feature map of the pseudo 3D convolutional layer to obtain spatial information. The formula used is as follows:
[0103] ;
[0104] In the formula, Represents spatial information. and Represent the height and width of the feature map respectively, represents the output feature map of the pseudo 3D convolutional layer, It means that the time step is selected from the output feature map of the pseudo 3D convolution layer. , the spatial position is The pixel value of
[0105] Step S322: Compress the number of channels using The convolution kernel convolves the spatial information and compresses the number of channels to the original , get the compressed feature map, the formula used is as follows:
[0106] ;
[0107] In the formula, represents the compressed feature map, express Convolution kernel;
[0108] Step S323: Perform temporal reasoning to reconstruct the compressed feature map into a feature map with the same temporal dimension as the output feature map of the pseudo 3D convolutional layer, and obtain the reconstructed feature map. The convolution kernel processes the reconstructed feature map to obtain the feature map after time inference. The formula used is as follows:
[0109] ;
[0110] In the formula, represents the feature map after temporal reasoning, represents the reconstructed feature map, express Convolution kernel;
[0111] Step S324: Obtain channel excitation matrix, using The convolution kernel and activation function process the feature map after temporal reasoning to obtain the channel excitation matrix. The formula used is as follows:
[0112] ;
[0113] ;
[0114] In the formula, Indicates the use The feature map obtained by the convolution kernel processing the feature map after temporal reasoning, express Convolution kernel, represents the channel excitation matrix, express Activation function;
[0115] Step S325: Calculate the output features of the channel attention layer. The formula used is as follows:
[0116] ;
[0117] In the formula, represents the output features of the channel attention layer, represents feature fusion operation;
[0118] Step S33: fusing the output features of the pseudo 3D convolution layer and the output features of the channel attention layer through the feature fusion layer to obtain the final output features;
[0119] Step S34: Introduce a multi-scale motion attention mechanism. Design a multi-scale motion attention mechanism in the multi-scale motion attention layer to construct explicit motion features. By calculating the feature differences between adjacent frames, pay attention to and extract the temporal feature information of the entire gesture motion process. The steps are as follows:
[0120] Step S341: Feature dimensionality reduction, the final output feature is Two-dimensional convolution is used to reduce the dimension, and we get ,in Indicates the final output feature at time step , and Feature map on ;
[0121] Step S342: Calculate the difference between two adjacent frames of the final output feature map to obtain a difference map. The calculation process is as follows:
[0122] ;
[0123] In the formula, Represents the time step The input feature map is Represents the time step The input feature map is express Convolution operation, represents a difference map;
[0124] Step S343: Process the difference graph and stack it according to the time dimension In order to deal with the position offset problem between the feature maps of two adjacent frames, we use Spatial convolution is used to fuse the position information of the feature maps of two adjacent frames and then spliced into the time dimension. The video data of the frame can only be generated Frame difference map, to ensure data integrity, the first Zero padding of frames to obtain a complete difference map;
[0125] Step S344: multi-scale pooling, input the complete difference map into the multi-scale pooling layer, perform downsampling operation through pooling kernels of different scales to obtain a multi-scale pooling difference map, and The activation function generates the motion attention weight coefficient;
[0126] Step S345: obtaining the output of the multi-scale motion attention layer, multiplying the attention weight coefficient by the final output feature element by element, and obtaining the output feature of the multi-scale motion attention layer;
[0127] Step S4: Model training and evaluation: Use the output features of the multi-scale motion attention layer to train the gesture recognition network to obtain the trained gesture recognition network. Use the test set to evaluate the performance of the trained gesture recognition network, output the probability score of the gesture action category, and select the highest probability score as the prediction result.
[0128] By performing the operations, in view of the problems of too many parameters and high computational complexity when general gesture recognition algorithms are deployed on mobile and embedded devices, this solution extracts spatiotemporal features from input features through a pseudo 3D residual convolution layer, and introduces a channel attention mechanism therein, which effectively extracts spatiotemporal feature information while reducing the complexity of the model, designs a multi-scale motion attention mechanism to construct explicit motion features, and helps the model better understand the dynamic information of gesture motion by calculating the feature differences between adjacent frames, thereby improving recognition accuracy and efficiency.
[0129] Example 4, see Figure 3 This embodiment is based on the above embodiment, and the development of a speech recognition model to recognize the user's speech command includes the following steps:
[0130] Step B1: Data collection and preprocessing, collecting audio samples including different intonations, performing audio segmentation, removing silent parts, applying audio enhancement techniques, enhancing audio samples, adopting audio synthesis strategies, creating synthetic audio samples, and labeling and annotating synthetic audio samples;
[0131] Step B2: Dataset division and feature extraction: divide the synthesized audio samples into training set and test set.
[0132] Use Mel spectrum analysis to extract features;
[0133] Step B3: Model construction, establish a speech recognition model. The model starts from the input layer, performs preliminary feature extraction through a one-dimensional convolution layer, introduces a batch normalization layer and a ReLU activation function, and is supplemented by a Dropout layer for regularization. Then it passes through two GRU layers. Each GRU layer is followed by a Dropout layer, a batch normalization layer, and an attention mechanism layer. At the end of the model, two output layers are designed: one uses a sigmoid activation function to predict existence, and the other uses a softmax activation function to distinguish categories, forming the final output of the model;
[0134] Step B4: Model training, using the training set to train the speech recognition model;
[0135] Step B5: Model tuning: Use a hybrid optimization algorithm to optimize the hyperparameters of the speech recognition model to obtain an optimized speech recognition model. The steps are as follows:
[0136] Step B51: Initialize parameters to determine the spider monkey population size, disturbance rate, local and global leader limits, maximum number of groups, and termination conditions;
[0137] Step B52: Evaluate the initial solution, assign an initial position to each spider monkey, and calculate the fitness value of each spider monkey using the fitness function. The formula used is as follows:
[0138] ;
[0139] In the formula, Indicates The fitness value of a spider monkey, Indicates The objective function value of the spider monkey is represents the absolute value function;
[0140] Step B53: Find local and global leaders, determine the local leader of each group according to the fitness value, and determine the global leader from all local leaders;
[0141] Step B54: In the local leader stage, the position of each spider monkey is updated according to the position of the local leader and the randomly selected spider monkey. The formula used is as follows:
[0142] ;
[0143] In the formula, Represents the updated spider monkey In the In the iteration The new location of the Indicates Spider monkeys in The current location of the dimension, Indicates The local leader of the group is The location of the dimension, Indicates that from A randomly selected spider monkey in the group The location of the dimension, Indicates from or A random number chosen uniformly in the range;
[0144] Step B55: In the global leader phase, the position of each spider monkey is updated according to the position of the global leader and the randomly selected spider monkey. The formula used is as follows:
[0145] ;
[0146] In the formula, The global optimal spider monkey is The location of the dimension;
[0147] Step B56: Use simulated annealing algorithm to search for the global optimal solution. The steps are as follows:
[0148] Step B561: Use the simulated annealing algorithm to search around the global optimal solution, generate a scaling factor, and calculate the temperature value. Generate a new solution based on the temperature value, and calculate the fitness value of the new solution to find the optimal fitness value in each iteration. The formula used is as follows:
[0149] ;
[0150] In the formula, represents the average fitness value of spider monkeys around the global optimal solution, represents the sum of the fitness values of all spider monkeys, represents the spider monkey population, represents the fitness value of each solution considered in the simulated annealing process;
[0151] Step B562: Compare the fitness value of the new solution with the current solution. If the new solution is better than the current solution, accept the new solution and increase the number of iterations.
[0152] Step B563: Check whether the number of iterations has reached the maximum number of iterations. If so, update the new solution, update the position of the spider monkey according to the new solution, pay attention to the constraints of the local and global leaders while updating the position, and determine whether the termination condition is met. If so, obtain the output and terminate the algorithm. If the termination condition is not met, go to step B52 to continue execution; if the maximum number of iterations is not reached, go to step B561 to continue execution;
[0153] Step B564: If the new solution is not as good as the current solution, the new solution is accepted or rejected according to the acceptance probability. The acceptance probability is calculated using the following formula:
[0154] ;
[0155] In the formula, represents the probability of acceptance, Indicates the difference between the new solution and the current solution;
[0156] Generate a random number between 0 and 1. If the random number is less than the acceptance probability, accept the new solution, increase the number of iterations, and go to step B563 to continue execution. Otherwise, reject the new solution, accept the current solution, and continue iterating, executing step B563.
[0157] Step B6: Model evaluation: Use the test set to evaluate the optimized speech recognition model, and use accuracy and F1 score as evaluation indicators to evaluate the model performance.
[0158] By performing the above operations, the hyperparameter search algorithm of a general speech recognition model is prone to fall into a local optimal solution and cannot find a global optimal solution, resulting in poor model performance and consumption of a large amount of computing resources. This solution uses the spider monkey algorithm to perform global and local searches to find local optimal solutions and global optimal solutions, and then uses the probability acceptance mechanism of the simulated annealing algorithm to perform local searches around the global optimal solution, further optimizing the quality of the solution, avoiding falling into the local optimal solution, and improving the optimization efficiency.
[0159] Embodiment 5: This embodiment is based on the above embodiment. The specific process of data collection and preprocessing in step B1 is as follows: male and female audio samples with different intonations are collected, each audio sample includes: "forward", "backward", and "jump". In addition, error audio samples and background audio are also collected. By setting the minimum silence length to 1500 milliseconds and the silence threshold to -50 decibels, key speech segments are extracted and the silence parts are removed. In order to expand the data set and enhance the generalization ability of the model, audio enhancement techniques are applied to the correct audio segments, including time stretching or compression (sampling from a uniform distribution) and pitch adjustment (in The correct audio, incorrect audio, and background audio are combined to create synthetic audio samples through 4000 iterations, and the synthetic audio samples are labeled and annotated.
[0160] Embodiment 6. This embodiment is based on the above embodiment. The specific process of data set division and feature extraction in step B2 is: 75% of the synthetic audio samples are divided into a training set and 25% are divided into a test set. Mel spectrum analysis is used for feature extraction. The FFT window size is set to 200, the window interval is 80 audio samples, 101 Mel filters are used, and the maximum frequency is limited to 8000 Hz.
[0161] Embodiment 7. This embodiment is based on the above embodiment. The hyperparameters of the speech recognition model in step B3 include: learning rate, maximum number of iterations, and batch size. After optimizing the speech recognition model using the hybrid optimization algorithm, the learning rate is set to 0.001, the maximum number of iterations is set to 100, and the batch size is set to 64.
[0162] Embodiment 8. This embodiment is based on the above embodiment. In step S344, the multi-scale pooling layer is composed of a maximum pooling layer with pooling kernel sizes of 2, 4 and 6 respectively. The multi-scale pooling layer enables features to be compressed from multiple dimensions and extracts pooling features of different scales, so that the network can learn feature information at different scales.
[0163] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
[0164] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.
Claims
1. An immersive paper shadow performance AR interactive system, characterized by: It includes paper shadow animation production module, virtual environment building module, and user interaction design module; The paper shadow animation production module collects paper shadow image materials, processes the collected paper shadow image materials, produces paper shadow animations through skeleton animation technology, and sends the paper shadow animations to the virtual environment construction module; The virtual environment building module uses the AR development platform to build a virtual environment, imports the paper shadow play animation as a virtual object into the virtual environment, and sends the virtual environment to the user interaction design module; The user interaction design module includes: a gesture recognition module, a voice recognition module, a command parsing module, a video playback module and a paper shadow control module; The gesture recognition module establishes a gesture recognition network to recognize the user's gesture state, and when a specific interactive gesture of the user is recognized, enters a corresponding processing flow according to the gesture type; The gesture recognition module establishes a gesture recognition network to recognize the user's gesture state, including the following steps: Step S1: data collection and preprocessing, collecting gesture video data, preprocessing the gesture video data, and obtaining preprocessed video data; Step S2: Data labeling: labeling the video frames in the preprocessed video data, clearly indicating the gesture category in each frame, and dividing it into a training set and a test set; Step S3: Establish a gesture recognition network. The gesture recognition network includes: a convolution layer, a pseudo 3D convolution layer, a channel attention layer, a feature fusion layer, and a multi-scale motion attention layer. The steps of constructing the gesture recognition network are as follows: Step S31: input the training set into a 1×1×1 convolutional layer, then pass through a pseudo 3D convolutional layer, and then pass through a 1×1×1 convolutional layer to obtain an output feature map of the pseudo 3D convolutional layer, wherein the pseudo 3D convolutional layer includes two convolutional layers: a 1×1×3 convolutional layer and a 3×3×1 convolutional layer; Step S32: Integrate the channel attention mechanism into the channel attention layer, model the channel relationship of the output feature map of the pseudo 3D convolution layer, obtain the channel information weight distribution of the feature, and obtain the output feature of the channel attention layer. The steps are as follows: Step S321: Acquire spatial information, perform spatial average pooling on the output feature map of the pseudo 3D convolutional layer, and obtain spatial information V; Step S322: compress the number of channels, use a 1×1 convolution kernel to convolve the spatial information, compress the number of channels to 1 / 16 of the original, and obtain a compressed feature map. The formula used is as follows: In r =N1*V; Where V r Represents the compressed feature map, N1 represents a 1×1 convolution kernel; Step S323: Perform temporal reasoning, reconstruct the compressed feature map into a feature map with the same temporal dimension as the output feature map of the pseudo 3D convolution layer, obtain a reconstructed feature map, process the reconstructed feature map through a 1×1 convolution kernel, and obtain a feature map after temporal reasoning. The formula used is as follows: In the formula, represents the feature map after temporal reasoning, represents the reconstructed feature map, N2 represents a 1×1 convolution kernel; Step S324: Obtain the channel excitation matrix, use a 1×1 convolution kernel and an activation function to process the feature map after temporal reasoning to obtain the channel excitation matrix. The formula used is as follows: C = σ(V0); Where V0 represents the feature map obtained by processing the feature map after temporal inference with a 1×1 convolution kernel, N3 represents a 1×1 convolution kernel, C represents the channel excitation matrix, and σ represents the Sigmoid activation function; Step S325: Calculate the output features of the channel attention layer. The formula used is as follows: out=V+V0⊙V; In the formula, out represents the output feature of the channel attention layer, ⊙ represents the feature fusion operation; Step S33: fusing the output features of the pseudo 3D convolution layer and the output features of the channel attention layer through the feature fusion layer to obtain the final output features; Step S34: Introduce a multi-scale motion attention mechanism. Design a multi-scale motion attention mechanism in the multi-scale motion attention layer to construct explicit motion features, pay attention to and extract the temporal feature information of the entire gesture motion process. The steps are as follows: Step S341: feature dimensionality reduction, the final output feature is reduced in dimension by 1×1 two-dimensional convolution to obtain D(t), D(t+1), and D(t+2), where D(t), D(t+1), and D(t+2) represent the feature maps of the final output feature at time steps t, t+1, and t+2; Step S342: Calculate the difference between two adjacent frames of the final output feature map to obtain a difference map. The calculation process is as follows: d t =Conv 3×3 (X t+1 )-X t ; Where, X t represents the input feature map at time step t, X t+1 Represents the input feature map at time step t+1, Conv 3×3 represents a 3×3 convolution operation, d t represents a difference map; Step S343: Process the difference map, stack t-1 difference maps according to the time dimension, use 3×3 spatial convolution to fuse the position information of the feature maps of two adjacent frames to deal with the position offset problem between the feature maps of two adjacent frames, and then splice them into the time dimension. Since the video data of t frames can only generate the difference map of t-1 frames, fill the tth frame with zeros to obtain a complete difference map; Step S344: multi-scale pooling, inputting the complete difference map into the multi-scale pooling layer, performing downsampling operation through pooling kernels of different scales to obtain the multi-scale pooling difference map, and generating the motion attention weight coefficient through the Softmax activation function; Step S345: obtaining the output of the multi-scale motion attention layer, multiplying the attention weight coefficient by the final output feature element by element, and obtaining the output feature of the multi-scale motion attention layer; Step S4: Model training and evaluation: Use the output features of the multi-scale motion attention layer to train the gesture recognition network to obtain the trained gesture recognition network. Use the test set to evaluate the performance of the trained gesture recognition network, output the probability score of the gesture action category, and select the highest probability score as the prediction result. The speech recognition module, when recognizing that the user is in a preset first gesture type, develops a speech recognition model, recognizes the user's voice command, and sends the user's voice command to the command parsing module; The command parsing module parses the user's voice command into specific operation instructions, and sends the specific operation instructions to the paper shadow control module; The paper shadow control module controls the paper shadow in the virtual environment to perform according to specific operation instructions; When the video playing module recognizes that the user has a preset second gesture type, the paper shadow science video is played.
2. The immersive paper shadow performance AR interactive system according to claim 1, characterized in that: When the speech recognition module recognizes that the user is in a preset first gesture type, a speech recognition model is developed to recognize the user's speech command, including the following steps: Step B1: Data collection and preprocessing; Step B2: Dataset partitioning and feature extraction; Step B3: Model construction; Step B4: model training; Step B5: Model tuning, including the following steps: Step B51: Initialize parameters to determine the spider monkey population size, disturbance rate, local and global leader limits, maximum number of groups, and termination conditions; Step B52: Evaluate the initial solution, assign an initial position to each spider monkey, and calculate the fitness value of each spider monkey using the fitness function; Step B53: Find local and global leaders, determine the local leader of each group according to the fitness value, and determine the global leader from all local leaders; Step B54: local leader stage, updating the position of each spider monkey according to the position of the local leader and the randomly selected spider monkey; Step B55: Global leader stage, updating the position of each spider monkey according to the position of the global leader and the randomly selected spider monkey; Step B56: Searching for the global optimal solution using a simulated annealing algorithm; Step B6: Model evaluation.
