Immersive travel experience system based on meta-universe technology

Abstract

The invention discloses an immersive travel experience system based on the meta-universe technology, which relates to the field of the meta-universe technology and comprises a scene construction module, an interaction equipment module, a data analysis module and a multi-mode interaction module. According to the immersive text travel experience system, virtual reality, augmented reality and 3D rendering technologies are integrated to construct a virtual text travel scene, special equipment is configured to enable a user to feel the virtual text travel scene personally on the scene, meanwhile, the system can further analyze user behavior data in real time, and the user experience is improved. The virtual scene is adjusted according to the interactive behavior of the user in the virtual travel scene, personalized touring experience is provided, the naturalness and depth of interaction are further enhanced by fusing gestures, voice interaction, eye movement tracking and electroencephalogram signal recognition technologies, the user can obtain richer and more real travel experience, and the user experience is improved. The immersion and individuation degree of cultural tourism are improved, and interesting tourism experience is brought.

Description

Technical Field

[0001] The present invention relates to the field of metaverse technology, and in particular to an immersive cultural tourism experience system based on metaverse technology. Background Art

[0002] With the continuous development of science and technology, Metaverse technology, as an emerging digital interactive platform, is gradually changing people's lifestyles and entertainment experiences. In the field of cultural tourism, traditional travel methods are often limited by geographical location, time and real resources, and it is difficult to meet people's needs for diversified and personalized travel experiences. Therefore, how to combine Metaverse technology to create a new immersive cultural and tourism experience system has become an urgent problem to be solved.

[0003] Traditional cultural tourism experience methods mostly rely on physical attractions and on-site visits. Tourists need to go to the destination in person to feel the local cultural atmosphere and natural landscape. However, this method not only consumes time and energy, but may also be affected by various factors such as weather and transportation. In addition, for some historical sites or specific cultural scenes, due to protection needs, tourists often find it difficult to get close or have an in-depth understanding. In order to solve the above problems, the industry has begun to explore the application of metaverse technology in the field of cultural tourism. Metaverse technology integrates virtual reality, augmented reality and 3D rendering technologies to build highly realistic virtual cultural tourism scenes, allowing tourists to experience the cultural characteristics and natural landscapes of different regions in an immersive way. However, the existing metaverse cultural tourism experience system still has some shortcomings, such as a single interactive method, lack of personalized experience, and insufficient scene construction. It is difficult to meet the needs of tourists for high-quality cultural tourism experience. In this regard, we propose an immersive cultural tourism experience system based on metaverse technology. Summary of the invention

[0004] In order to solve the above technical problems, an immersive cultural and tourism experience system based on metaverse technology is provided. This technical solution solves the above problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] An immersive cultural tourism experience system based on metaverse technology, including: a scene construction module, an interactive device module, a data analysis module and a multimodal interaction module;

[0007] The scene construction module is used to construct virtual cultural tourism scenes by integrating virtual reality, augmented reality and 3D rendering technology;

[0008] The interactive device module is electrically connected to the scene construction module, and the interactive device module is used to configure a dedicated device so that the user can enter and experience the virtual cultural and tourism scene with the help of the device to obtain an immersive tourism experience;

[0009] The data analysis module is electrically connected to the interactive device module, and the data analysis module is used to analyze the data in real time, adjust the virtual scene according to the user's interactive behavior in the virtual cultural tourism scene, and provide a personalized tour experience;

[0010] The multimodal interaction module is electrically connected to the data analysis module, and is used to enhance the naturalness and depth of the interaction through gesture and voice interaction, and integrates eye tracking and EEG signal recognition technology.

[0011] Preferably, the integration of virtual reality and augmented reality specifically includes:

[0012] In three-dimensional space, let the coordinates of an object in the real scene be (x r ,y r ,z r ), the coordinates of the virtual element's expected overlay position are (x v ,y v ,z v );

[0013] Use the space transformation matrix for conversion, the conversion formula is:

[0014]

[0015] In the formula, R represents the rotation matrix, and t represents the translation vector;

[0016] The final coordinates of the fused virtual elements in the real scene are obtained by calculation, where the final coordinate calculation formula is:

[0017] x f ,y f ,z f ,1] T =T×[x v ,y v ,z v ,1] T

[0018] In the formula, (x f ,y f ,z f ) represents the final coordinates in the real scene. Through the spatial positioning algorithm, the virtual elements can be accurately superimposed on the corresponding positions in the real scene, realizing the precise integration of virtual elements and the real scene.

[0019] Preferably, the 3D rendering technology specifically includes:

[0020] For each pixel in the scene, shoot a ray from the viewpoint:

[0021] B=O+τD

[0022] In the formula, B represents the position vector of the light in space, O represents the origin of the light, that is, the viewpoint position, D represents the unit vector of the light direction, and τ represents the propagation distance of the light;

[0023] When light intersects the surface of an object in the scene, the color of the pixel is determined by calculating the lighting conditions at the intersection, where the calculation formula is:

[0024]

[0025] Where I represents the intensity of incident light, k d Represents the diffuse reflectance, I a Indicates the ambient light intensity, I li represents the intensity of the i-th point light source, represents the surface normal vector of the object, Represents the direction vector from the intersection point to the i-th point light source;

[0026] When the user interacts with the scene, the collision detection algorithm is used to determine whether the user operation intersects with the scene objects, so as to implement the operation on the scene elements. The specific collision detection algorithm is:

[0027] Assume that there are two objects in the scene, each represented by its axis-aligned bounding box. For object A, the minimum coordinate point of its bounding box is (min Ax ,min Ay ,min Az ), the maximum coordinate point is (max Ax ,max Ay ,max Az ), for object B, the minimum coordinate point of its bounding box is (min Bx ,min By ,min Bz ), the maximum coordinate point is (max Bx ,max By ,max Bz );

[0028] The condition for detecting whether two axis-aligned bounding boxes intersect is:

[0029]

[0030] When the above three conditions are met at the same time, it means that the axis-aligned bounding boxes of the two objects intersect, that is, the user operation collides with the scene object.

[0031] Preferably, the data analysis module is used for real-time data analysis and specifically includes:

[0032] Collecting user behavior data based on the interactive device module, the user behavior data including: the duration of stay at the virtual scenic spot, the movement speed and operation frequency within the virtual scenic spot;

[0033] Based on the obtained user behavior data, define the user's interest level:

[0034]

[0035] In the formula, S represents the user's interest, α 1 , α 2 , α 3 They represent the weight coefficients of residence time, movement speed and operation frequency, t A represents the duration of stay at attraction A, T total represents the total tour time, v avg represents the average moving speed, v max represents the maximum moving speed, and f represents the operating frequency;

[0036] Based on the calculated user interest and the association matrix between attractions, the next attraction is recommended through an algorithm and the scene content is adjusted to achieve a personalized tour experience.

[0037] Preferably, the method for constructing the association matrix between scenic spots is:

[0038] Assume that there are m scenic spots in the system, denoted by A 1 ,A 2 ,…,A m ;

[0039] Construct an m×m matrix M, where M ij Indicates attraction A i With attractions A j The degree of association is calculated by ij The steps are:

[0040] Collect a large number of users' tour path data and count the number of users from attraction A to attraction B. i To attraction A j The number of transfers;

[0041] For each attraction, calculate the total number of transfers from that attraction:

[0042]

[0043] In the formula, total i Indicates the total number of transfers, count ij From attraction A i To attraction A j The number of transfers;

[0044] Attraction Ai With attractions A j The degree of association is updated to:

[0045]

[0046] The obtained matrix M ij The larger the value, the more scenic spot A i With attractions A j The stronger the correlation.

[0047] Preferably, the method for recommending the next scenic spot through the algorithm is:

[0048] Based on the user's interest and the attraction association matrix, the recommendation score of each attraction is calculated:

[0049] score j =S×M kj

[0050] In the formula, score j represents the recommendation score of the scenic spot, M kj represents an element in the scenic spot association matrix M, k represents the index of the scenic spot the user is currently in, and j represents the indexes of other scenic spots;

[0051] Sort the recommendation scores of all attractions and select the attraction with the highest score as the next recommended attraction.

[0052] Preferably, the gesture interaction is implemented using computer vision technology;

[0053] The implementation method is:

[0054] The user gesture is extracted and the Hu moment is used as the feature descriptor. For the binarized gesture image IM(x,y), its p+q order geometric moment is defined as:

[0055]

[0056] In the formula, R pq represents a p+q order geometric moment, x and y represent coordinates in the image coordinate system, p and q represent non-negative integers, which are used to determine the order of the geometric moment;

[0057] Its p+q central moment is defined as:

[0058]

[0059] In the formula, μ pq represents the p+q order central moment;

[0060] Normalized central moment:

[0061]

[0062] in, p+q≥2;

[0063] The Hu moment is composed of a linear combination of normalized central moments. By matching with the predefined gesture template Hu moment, the Euclidean distance between the gesture to be recognized and the template gesture is calculated. When the Euclidean distance is less than the set threshold, it is recognized as the corresponding gesture command, thereby controlling the virtual scene operation. The Euclidean distance calculation formula is:

[0064]

[0065] In the formula, θ represents the Euclidean distance, represents the i-th Hu moment feature of the gesture to be recognized, Represents the i-th Hu moment feature of the template gesture.

[0066] Preferably, the voice interaction is implemented using speech recognition and natural language processing technology;

[0067] In the speech recognition stage, for the input speech signal θ = θ 1 ,θ 1 ,…,θ T , the HMM model consists of a state set E = {E 1 ,E 1 ,…,E N}, the initial state probability vector π=(π 1 ,π 1 ,…,π N ), the state transition probability matrix G = {G ij}, the observed value probability distribution H = {h j (θ t )}constitute;

[0068] The probability of the speech signal under the model is calculated through the forward-backward algorithm, and the word sequence corresponding to the model with the highest probability is selected as the recognition result;

[0069] In the natural language processing stage, the recognized text is subjected to lexical analysis. A rule-based part-of-speech tagging method is used to tag each word with the part-of-speech tagging rule library. Then, syntactic analysis is performed and a dependency syntax tree is constructed. By analyzing the dependency relationship between sentence components, the user's semantics is understood, thereby achieving effective voice interaction with the system.

[0070] Preferably, the eye tracking method is:

[0071] According to the principle of corneal reflection, let the reflection point of the infrared light emitted by the eye tracker on the cornea be C, the center of the eyeball be Z, and the edge of the pupil be U;

[0072] By measuring ∠CZU and the position parameters of the eyeball in the eye socket, the sight direction vector is calculated using geometric relationships.

[0073] In the virtual scene, for the scene element W, its position vector is The viewpoint position vector is

[0074] like If a scene element is within the field of vision, the system determines that the user is looking at the element. At this time, the system queries the database for a detailed introduction of the element and automatically pops up the introduction, without the need for manual operation by the user, thus improving the convenience of interaction.

[0075] Preferably, the method for recognizing EEG signals is:

[0076] Preprocess the collected EEG signals and filter to remove noise;

[0077] Based on the feature extraction of the preprocessed EEG signal, its power spectral density estimation P xx (f) Calculated by Welch method:

[0078]

[0079] In the formula, K represents the number of segments, and N represents the length of each segment of data;

[0080] By analyzing the power spectral density characteristics of different frequency bands, using the support vector machine classification algorithm, taking the EEG signal feature vector as input, the trained SVM model is used to identify the user's emotional state, and then push the corresponding experience content. The expression of the SVM model is:

[0081]

[0082] In the formula, α i represents the Lagrange multiplier, y i represents the sample label, K(x i ,x) represents the kernel function, and b represents the bias.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] The immersive cultural tourism experience system proposed in the present invention constructs a virtual cultural tourism scene by integrating virtual reality, augmented reality and 3D rendering technology, and configures special equipment to enable users to experience the virtual cultural tourism scene in an immersive way. At the same time, the system can also analyze user behavior data in real time, adjust the virtual scene according to the user's interactive behavior in the virtual cultural tourism scene, and provide a personalized tour experience. By integrating gestures, voice interaction, eye tracking, and EEG signal recognition technology, the naturalness and depth of the interaction are further enhanced, so that users can obtain a richer and more realistic travel experience, which enhances the immersion and personalization of cultural tourism and brings tourists a more novel and interesting travel experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 This is a module framework diagram of the immersive cultural and tourism experience system based on metaverse technology. DETAILED DESCRIPTION

[0086] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0087] Reference Figure 1 As shown, an immersive cultural tourism experience system based on metaverse technology includes: a scene construction module, an interactive device module, a data analysis module and a multimodal interaction module;

[0088] The scene construction module gives full play to the advantages of virtual reality, augmented reality and 3D rendering technology. Through the detailed scanning and data collection of real-world cultural and tourism scenes, combined with rich historical and cultural materials and creative imagination, and using powerful 3D rendering capabilities, a highly realistic and immersive virtual cultural and tourism scene is constructed. Whether it is the magnificent Forbidden City complex or the mysterious and fantastic underwater world, it can be presented vividly.

[0089] The interactive device module is closely connected with the scene construction module, and is carefully configured with special equipment such as head-mounted displays, tactile feedback gloves, motion capture suits, etc. These devices accurately capture the user's movements and postures, synchronize them with the virtual scene, and provide users with multi-dimensional sensory feedback such as vision, hearing, and touch, allowing users to feel as if they are really stepping into the virtual cultural and tourism scene and fully appreciate the wonders in it.

[0090] The data analysis module monitors and deeply analyzes various interactive data of users in the virtual cultural tourism scene in real time, such as walking paths, staying areas, operation frequency, attention to different attractions, etc. Based on these detailed data, the virtual scene content is intelligently adjusted, such as adding relevant historical allusions for users interested in history, and optimizing the natural landscape details in the scene for users who prefer natural scenery, creating a personalized tour experience.

[0091] The multimodal interaction module uses gesture recognition technology to allow users to interact with the virtual environment through simple and natural gestures, such as waving to summon a virtual tour guide and reaching out to touch virtual exhibits. The voice interaction function enables real-time dialogue between users and scenes, and it is convenient and efficient to query information and issue instructions. Integrating eye tracking technology, the system can capture the user's focus and display relevant content first; EEG signal recognition technology further perceives the user's emotional state, dynamically adjusts the scene atmosphere and interaction method, and enhances the naturalness and depth of interaction.

[0092] Providing users with a highly personalized and immersive cultural and tourism experience that transcends time and space constraints has greatly enhanced the fun and appeal of cultural and tourism consumption. It helps the cultural and tourism industry break through traditional display and experience models, expand business boundaries, create new development opportunities, and promote the digital and intelligent transformation and upgrading of the cultural and tourism industry.

[0093] The integration of virtual reality and augmented reality specifically includes:

[0094] In three-dimensional space, let the coordinates of an object in the real scene be (x r ,y r ,z r ), the coordinates of the virtual element's expected overlay position are (x v ,y v ,z v );

[0095] Use the space transformation matrix for conversion, the conversion formula is:

[0096]

[0097] In the formula, R represents the rotation matrix, and t represents the translation vector;

[0098] The final coordinates of the fused virtual elements in the real scene are obtained by calculation, where the final coordinate calculation formula is:

[0099] [x f ,y f ,z f ,1] T =T×[x v ,y v ,z v ,1] T

[0100] In the formula, (x f ,y f ,z f ) represents the final coordinates in the real scene. Through the spatial positioning algorithm, the virtual elements can be accurately superimposed on the corresponding positions in the real scene, realizing the precise integration of virtual elements and the real scene.

[0101] This technology that realizes the integration of virtual reality and augmented reality has obvious advantages due to its precise coordinate setting and spatial transformation calculation. Through rigorous rotation matrix and translation vector calculation, the coordinates of virtual elements can be accurately converted to real scenes. In this way, virtual elements can be seamlessly integrated into real scenes without any unreality such as dislocation and floating. In the cultural and tourism experience scene, it allows tourists to see virtual historical figures and scene reproductions that fit the actual layout in a real environment, so that the tour is no longer limited to the real landscape, greatly enriching the experience content, deepening the sense of immersion, and bringing tourists a unique and in-depth cultural and tourism experience.

[0102] The 3D rendering technology specifically includes:

[0103] For each pixel in the scene, shoot a ray from the viewpoint:

[0104] B=O+τD

[0105] In the formula, B represents the position vector of the light in space, O represents the origin of the light, that is, the viewpoint position, D represents the unit vector of the light direction, and τ represents the propagation distance of the light;

[0106] When light intersects the surface of an object in the scene, the color of the pixel is determined by calculating the lighting conditions at the intersection, where the calculation formula is:

[0107]

[0108] Where I represents the intensity of incident light, k d Represents the diffuse reflectance, I a Indicates the ambient light intensity, I li represents the intensity of the i-th point light source, represents the surface normal vector of the object, Represents the direction vector from the intersection point to the i-th point light source;

[0109] When the user interacts with the scene, the collision detection algorithm is used to determine whether the user operation intersects with the scene objects, so as to implement the operation on the scene elements. The specific collision detection algorithm is:

[0110] Assume that there are two objects in the scene, each represented by its axis-aligned bounding box. For object A, the minimum coordinate point of its bounding box is (min Ax ,min Ay ,min Az ), the maximum coordinate point is (max Ax ,max Ay ,max Az ), for object B, the minimum coordinate point of its bounding box is (min Bx ,min By ,minBz ), the maximum coordinate point is (max Bx ,max By ,max Bz );

[0111] The condition for detecting whether two axis-aligned bounding boxes intersect is:

[0112]

[0113] When the above three conditions are met at the same time, it means that the axis-aligned bounding boxes of the two objects intersect, that is, the user operation collides with the scene object.

[0114] The calculation of pixel color by emitting light from the viewpoint can accurately simulate the propagation and reflection of light in the scene, making the rendered scene lighting effect extremely realistic. Whether it is soft natural light or bright artificial light, it can be presented delicately, bringing an immersive visual experience to users. The collision detection algorithm simply compares the coordinates of the axis-aligned bounding box to efficiently determine whether the user operation intersects with the scene objects, ensuring that the user's interaction with the scene elements is natural and smooth. For example, operations such as touching exhibits and pushing objects can respond accurately, greatly enhancing the authenticity and fun of the interaction, and comprehensively improving the quality of the user's experience in the virtual scene.

[0115] The data analysis module is used for real-time data analysis, including:

[0116] Collecting user behavior data based on the interactive device module, the user behavior data including: the duration of stay at the virtual scenic spot, the movement speed and operation frequency within the virtual scenic spot;

[0117] Based on the obtained user behavior data, define the user's interest level:

[0118]

[0119] In the formula, S represents the user's interest, α 1 , α 2 , α 3 They represent the weight coefficients of residence time, movement speed and operation frequency, t A represents the duration of stay at attraction A, T total represents the total tour time, v avg represents the average moving speed, v max represents the maximum moving speed, and f represents the operating frequency;

[0120] Based on the calculated user interest and the association matrix between attractions, the next attraction is recommended through an algorithm and the scene content is adjusted to achieve a personalized tour experience.

[0121] Based on the user's stay time, movement speed and operation frequency at the virtual attractions, the interest level is accurately calculated. Then, combined with the attraction association matrix, attractions are recommended, scenes are adjusted, and tour routes are tailored for users, greatly improving the personalized experience, allowing each user to deeply explore the content of interest and enjoy a unique virtual cultural and tourism journey.

[0122] The method for constructing the association matrix between scenic spots is:

[0123] Assume that there are m scenic spots in the system, denoted by A 1 ,A 2 ,…,A m ;

[0124] Construct an m×m matrix M, where M ij Indicates attraction A i With attractions A j The degree of association is calculated by ij The steps are:

[0125] Collect a large number of users' tour path data and count the number of users from attraction A to attraction B. i To attraction A j The number of transfers;

[0126] For each attraction, calculate the total number of transfers from that attraction:

[0127]

[0128] In the formula, total i Indicates the total number of transfers, count ij From attraction A i To attraction A j The number of transfers;

[0129] Attraction A i With attractions A j The degree of association is updated to:

[0130]

[0131] The obtained matrix M ij The larger the value, the more scenic spot A i With attractions A j The stronger the correlation.

[0132] This method of constructing an attraction association matrix can accurately quantify the degree of association between attractions by collecting a large amount of user tour path data. It can reflect the connection between attractions based on real user behavior, provide strong support for personalized recommendations, help the system to more reasonably recommend attractions based on user interests, optimize tour routes, enhance tourists' tour experience, and help the immersive cultural and tourism experience system better serve users.

[0133] The method of recommending the next attraction through the algorithm is:

[0134] Based on the user's interest and the attraction association matrix, the recommendation score of each attraction is calculated:

[0135] score j =S×M kj

[0136] In the formula, score j represents the recommendation score of the scenic spot, M kj represents an element in the scenic spot association matrix M, k represents the index of the scenic spot the user is currently in, and j represents the indexes of other scenic spots;

[0137] Sort the recommendation scores of all attractions and select the attraction with the highest score as the next recommended attraction.

[0138] The recommendation score is calculated by combining the user's interest and the attraction association matrix, which comprehensively considers the user's personal preferences and the general connection between attractions. The recommended attractions are selected according to the score sorting, which can accurately push attractions that match the user's interests, optimize the tour experience, avoid tourists' blind exploration, and create an efficient and personalized virtual cultural travel itinerary for them.

[0139] The gesture interaction is implemented using computer vision technology;

[0140] The implementation method is:

[0141] The user gesture is extracted and the Hu moment is used as the feature descriptor. For the binarized gesture image IM(x,y), its p+q order geometric moment is defined as:

[0142]

[0143] In the formula, R pq represents a p+q order geometric moment, x and y represent coordinates in the image coordinate system, p and q represent non-negative integers, which are used to determine the order of the geometric moment;

[0144] Its p+q central moment is defined as:

[0145]

[0146] In the formula, μ pq represents the p+q order central moment;

[0147] Normalized central moment:

[0148]

[0149] in, p+q≥2;

[0150] The Hu moment is composed of a linear combination of normalized central moments. By matching with the predefined gesture template Hu moment, the Euclidean distance between the gesture to be recognized and the template gesture is calculated. When the Euclidean distance is less than the set threshold, it is recognized as the corresponding gesture command, thereby controlling the virtual scene operation. The Euclidean distance calculation formula is:

[0151]

[0152] In the formula, θ represents the Euclidean distance, represents the i-th Hu moment feature of the gesture to be recognized, Represents the i-th Hu moment feature of the template gesture.

[0153] Based on computer vision technology, there are many benefits to using Hu moments to achieve gesture interaction. By extracting the geometric moment, central moment and normalized central moment of the gesture image to form the Hu moment, the unique characteristics of the gesture can be effectively captured. By matching the Hu moment with the predefined gesture template and calculating the Euclidean distance, the gesture can be accurately recognized. This method does not require users to use complex equipment. With simple gestures, they can naturally control the virtual scene. The operation is intuitive and convenient, which greatly enhances the fluency and fun of the user's interaction with the virtual scene and improves the overall quality of the immersive cultural and tourism experience.

[0154] The voice interaction is achieved by using voice recognition and natural language processing technology;

[0155] In the speech recognition stage, for the input speech signal θ = θ 1 ,θ 1 ,…,θ T , the HMM model consists of a state set E = {E 1 ,E 1 ,...,E N}, the initial state probability vector π=(π 1 ,π 1 ,…,π N ), the state transition probability matrix G = {G ij}, the observed value probability distribution H = {h j (θ t )}constitute;

[0156] The probability of the speech signal under the model is calculated through the forward-backward algorithm, and the word sequence corresponding to the model with the highest probability is selected as the recognition result;

[0157] In the natural language processing stage, the recognized text is subjected to lexical analysis. A rule-based part-of-speech tagging method is used to tag each word with the part-of-speech tagging rule library. Then, syntactic analysis is performed and a dependency syntax tree is constructed. By analyzing the dependency relationship between sentence components, the user's semantics is understood, thereby achieving effective voice interaction with the system.

[0158] This voice interaction based on speech recognition and natural language processing has greatly improved the interaction efficiency and experience. Speech recognition uses the HMM model and forward-backward algorithm to accurately recognize speech, and natural language processing understands semantics through lexical and syntactic analysis. Users can communicate directly by voice, free from the constraints of manual operation, and the system responds quickly and accurately, making the cultural and tourism experience more convenient and smooth, and enhancing the sense of immersion.

[0159] Eye tracking methods are:

[0160] According to the principle of corneal reflection, let the reflection point of the infrared light emitted by the eye tracker on the cornea be C, the center of the eyeball be Z, and the edge of the pupil be U;

[0161] By measuring ∠CZU and the position parameters of the eyeball in the eye socket, the sight direction vector is calculated using geometric relationships.

[0162] In the virtual scene, for the scene element W, its position vector is The viewpoint position vector is

[0163] like If a scene element is within the field of vision, the system determines that the user is looking at the element. At this time, the system queries the database for a detailed introduction of the element and automatically pops up the introduction, without the need for manual operation by the user, thus improving the convenience of interaction.

[0164] Eye tracking technology uses the principle of corneal reflection to accurately calculate the direction of the line of sight, and can quickly determine the element that the user is looking at in the virtual scene. Once the gaze is determined and the element is within the line of sight, the system automatically pops up a detailed introduction without manual operation, greatly improving the convenience of interaction, allowing users to focus more on the scene experience, seamlessly obtain information, and enhance the fluency and fun of the immersive cultural and tourism experience.

[0165] The method of EEG signal recognition is:

[0166] Preprocess the collected EEG signals and filter to remove noise;

[0167] Based on the feature extraction of the preprocessed EEG signal, its power spectral density estimation P xx (f) Calculated by Welch method:

[0168]

[0169] In the formula, K represents the number of segments, and N represents the length of each segment of data;

[0170] By analyzing the power spectral density characteristics of different frequency bands, using the support vector machine classification algorithm, taking the EEG signal feature vector as input, the trained SVM model is used to identify the user's emotional state, and then push the corresponding experience content. The expression of the SVM model is:

[0171]

[0172] In the formula, α i represents the Lagrange multiplier, y i represents the sample label, K(x i ,x) represents the kernel function, and b represents the bias.

[0173] The EEG signal recognition method has outstanding advantages. First, pre-process and filter out noise, then use the Welch method to extract power spectrum density features, and accurately identify user emotions through the SVM algorithm. According to the emotion, the corresponding experience content is pushed, which can deeply fit the user's psychological state and customize an immersive cultural and tourism experience that better meets their needs, greatly improving the pertinence and appeal of the experience.

[0174] The use process of the present invention is as follows: the user enters the virtual cultural and tourism scene constructed by the scene construction module through the interactive device module, and experiences an immersive tourism experience with the help of virtual reality and augmented reality technology. During the tour, the data analysis module collects user behavior data in real time, and recommends the next attraction based on the user's interest and the attraction association matrix to achieve personalized tours. At the same time, the multimodal interaction module supports multiple interaction methods such as gestures, voice, eye tracking, and EEG signal recognition, allowing users to interact with virtual scenes naturally and conveniently, and obtain a richer and deeper cultural and tourism experience.

[0175] In summary, the advantages of the present invention are: the system provides users with an immersive travel experience by integrating virtual reality, augmented reality and 3D rendering technology, breaking the geographical and time limitations of traditional travel methods. The system can analyze user behavior data in real time, and provide personalized tour route recommendations based on user interest and scenic spot association matrix, thereby enhancing the fun and interactivity of the tour. At the same time, the multimodal interaction module integrates gesture, voice, eye tracking and EEG signal recognition technology, significantly improving the naturalness and depth of the interaction, allowing users to interact with virtual scenes more conveniently and intuitively, enriching the user's cultural and travel experience.

[0176] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. An immersive cultural tourism experience system based on metaverse technology, characterized in that: include: Scenario construction module, interactive device module, data analysis module and multimodal interaction module; The scene construction module is used to construct virtual cultural tourism scenes by integrating virtual reality, augmented reality and 3D rendering technology; The interactive device module is electrically connected to the scene construction module, and the interactive device module is used to configure a dedicated device so that the user can enter and experience the virtual cultural and tourism scene with the help of the device to obtain an immersive tourism experience; The data analysis module is electrically connected to the interactive device module, and the data analysis module is used to analyze the data in real time, adjust the virtual scene according to the user's interactive behavior in the virtual cultural tourism scene, and provide a personalized tour experience; The multimodal interaction module is electrically connected to the data analysis module, and is used to enhance the naturalness and depth of the interaction through gesture and voice interaction, and integrates eye tracking and EEG signal recognition technology.

2. According to claim 1, an immersive cultural tourism experience system based on metaverse technology is characterized in that: The integration of virtual reality and augmented reality specifically includes: In three-dimensional space, let the coordinates of an object in the real scene be (x r ,y r , z r ), the coordinates of the virtual element's expected overlay position are (x v ,y v , z v ); Use the space transformation matrix for conversion, the conversion formula is: In the formula, R represents the rotation matrix, and t represents the translation vector; The final coordinates of the fused virtual elements in the real scene are obtained by calculation, where the final coordinate calculation formula is: [x f ,y f ,z f ,1] T =T×[x v ,y v ,z v ,1] T In the formula, (x f ,y f , z f ) represents the final coordinates in the real scene. Through the spatial positioning algorithm, the virtual elements can be accurately superimposed on the corresponding positions in the real scene, realizing the precise integration of virtual elements and the real scene.

3. The immersive cultural tourism experience system based on metaverse technology according to claim 1 is characterized in that: The 3D rendering technology specifically includes: For each pixel in the scene, shoot a ray from the viewpoint: B=O+τD In the formula, B represents the position vector of the light in space, O represents the origin of the light, that is, the viewpoint position, D represents the unit vector of the light direction, and τ represents the propagation distance of the light; When light intersects the surface of an object in the scene, the color of the pixel is determined by calculating the lighting conditions at the intersection, where the calculation formula is: Where I represents the intensity of incident light, k d Represents the diffuse reflectance, I a Indicates the ambient light intensity, I li represents the intensity of the i-th point light source, represents the surface normal vector of the object, Represents the direction vector from the intersection point to the i-th point light source; When the user interacts with the scene, the collision detection algorithm is used to determine whether the user operation intersects with the scene objects, so as to implement the operation on the scene elements. The specific collision detection algorithm is: Assume that there are two objects in the scene, each represented by its axis-aligned bounding box. For object A, the minimum coordinate point of its bounding box is (min Ax ,min Ay ,min Az ), the maximum coordinate point is (max Ax ,max Ay ,max Az ), for object B, the minimum coordinate point of its bounding box is (min Bx ,min By , min Bz ), the maximum coordinate point is (max Bx ,max By , max Bz ); The condition for detecting whether two axis-aligned bounding boxes intersect is: When the above three conditions are met at the same time, it means that the axis-aligned bounding boxes of the two objects intersect, that is, the user operation collides with the scene object.

4. The immersive cultural tourism experience system based on metaverse technology according to claim 1 is characterized in that: The data analysis module is used for real-time data analysis, including: Collecting user behavior data based on the interactive device module, the user behavior data including: the duration of stay at the virtual scenic spot, the movement speed and operation frequency within the virtual scenic spot; Based on the obtained user behavior data, define the user's interest level: In the formula, S represents the user's interest, α1, α2, and α3 represent the weight coefficients of residence time, movement speed, and operation frequency, respectively, and t A represents the duration of stay at attraction A, T total represents the total tour time, v avg represents the average moving speed, v max represents the maximum moving speed, and f represents the operating frequency; Based on the calculated user interest and the association matrix between attractions, the next attraction is recommended through an algorithm and the scene content is adjusted to achieve a personalized tour experience.

5. The immersive cultural tourism experience system based on metaverse technology according to claim 4 is characterized in that: The method for constructing the association matrix between scenic spots is: Assume that there are m scenic spots in the system, denoted by A1, A2, ..., A m ; Construct an m×m matrix M, where M ij Indicates attraction A i With attractions A j The degree of association is calculated by ij The steps are: Collect a large number of users' tour path data and count the number of users from attraction A to attraction B. i To attraction A j The number of transfers; For each attraction, calculate the total number of transfers from that attraction: In the formula, total i Indicates the total number of transfers, count ij From attraction A i To attraction A j The number of transfers; Attraction A i With attractions A j The degree of association is updated to: The obtained matrix M ij The larger the value, the more scenic spot A i With attractions A j The stronger the correlation.

6. The immersive cultural tourism experience system based on metaverse technology according to claim 4 is characterized in that: The method of recommending the next attraction through the algorithm is: Based on the user's interest and the attraction association matrix, the recommendation score of each attraction is calculated: score j =S×M kj In the formula, score j represents the recommendation score of the scenic spot, M kj represents an element in the scenic spot association matrix M, k represents the index of the scenic spot the user is currently in, and j represents the indexes of other scenic spots; Sort the recommendation scores of all attractions and select the attraction with the highest score as the next recommended attraction.

7. The immersive cultural tourism experience system based on metaverse technology according to claim 1 is characterized in that: The gesture interaction is implemented using computer vision technology; The implementation method is: The user gesture is extracted and the Hu moment is used as the feature descriptor. For the binarized gesture image IM(x, y), its p+q order geometric moment is defined as: In the formula, R pq represents a p+q order geometric moment, x and y represent coordinates in the image coordinate system, p and q represent non-negative integers, which are used to determine the order of the geometric moment; Its p+q central moment is defined as: In the formula, μ pq represents the p+q order central moment; Normalized central moment: in, p+q≥2; The Hu moment is composed of a linear combination of normalized central moments. By matching with the predefined gesture template Hu moment, the Euclidean distance between the gesture to be recognized and the template gesture is calculated. When the Euclidean distance is less than the set threshold, it is recognized as the corresponding gesture command, thereby controlling the virtual scene operation. The Euclidean distance calculation formula is: In the formula, θ represents the Euclidean distance, represents the i-th Hu moment feature of the gesture to be recognized, Represents the i-th Hu moment feature of the template gesture.

8. The immersive cultural tourism experience system based on metaverse technology according to claim 1 is characterized in that: The voice interaction is achieved by using voice recognition and natural language processing technology; In the speech recognition stage, for the input speech signal θ=θ1,θ1,…,θ T , the HMM model consists of a state set E = {E1, E1, ..., E N }, initial state probability vector π = (π1, π1, ..., π N ), the state transition probability matrix G = {G ij }, the observed value probability distribution H = {h j (θ t )}constitute; The probability of the speech signal under the model is calculated through the forward-backward algorithm, and the word sequence corresponding to the model with the highest probability is selected as the recognition result; In the natural language processing stage, the recognized text is subjected to lexical analysis. A rule-based part-of-speech tagging method is used to tag each word with the part-of-speech tagging rule library. Then, syntactic analysis is performed and a dependency syntax tree is constructed. By analyzing the dependency relationship between sentence components, the user's semantics is understood, thereby achieving effective voice interaction with the system.

9. The immersive cultural tourism experience system based on metaverse technology according to claim 1 is characterized in that: Eye tracking methods are: According to the principle of corneal reflection, let the reflection point of the infrared light emitted by the eye tracker on the cornea be C, the center of the eyeball be Z, and the edge of the pupil be U; By measuring ∠CZU and the position parameters of the eyeball in the eye socket, the sight direction vector is calculated using geometric relationships. In the virtual scene, for the scene element W, its position vector is The viewpoint position vector is like If a scene element is within the field of vision, the system determines that the user is looking at the element. At this time, the system queries the database for a detailed introduction of the element and automatically pops up the introduction, without the need for manual operation by the user, thus improving the convenience of interaction.

10. The immersive cultural tourism experience system based on metaverse technology according to claim 1, characterized in that: The method of EEG signal recognition is: Preprocess the collected EEG signals and filter to remove noise; Based on the feature extraction of the preprocessed EEG signal, its power spectral density estimation P xx (f) Calculated by Welch method: In the formula, K represents the number of segments, and N represents the length of each segment of data; By analyzing the power spectral density characteristics of different frequency bands, using the support vector machine classification algorithm, taking the EEG signal feature vector as input, the trained SVM model is used to identify the user's emotional state, and then push the corresponding experience content. The expression of the SVM model is: In the formula, α i represents the Lagrange multiplier, y i represents the sample label, K(x i , x) represents the kernel function, and b represents the bias.

Images