Virtual reality tour guide system

Abstract

The present invention relates to the field of virtual reality technology, and further to a virtual reality tour guide system. The system includes: an image acquisition unit, which is used to capture a panoramic image of the environment on the tour guide route to obtain a panoramic image sequence; preprocessing the panoramic image sequence to obtain a preprocessed panoramic image sequence; a feature extraction unit, which is used to obtain a feature point set; a path planning unit, which is used to plan the tour guide route; a feature matching unit, which is used to obtain a matching pair; a posture estimation unit, which is used to estimate the camera posture; a virtual model loading unit, which is used to load a virtual model to obtain a virtual model set; a scenic spot rendering unit, which is used to perform scenic spot rendering to obtain a rendered scenic spot image; a user experience unit, which is used to present the rendered scenic spot image to the user through an immersive display device. The present invention realizes rich virtual model loading and realistic scenic spot rendering, providing users with an immersive tour guide experience.

Description

Technical Field

[0001] The invention belongs to the technical field of virtual reality, and in particular relates to a virtual reality tour guide system. Background Art

[0002] With the rapid development of the tourism industry, more and more people choose to travel and sightsee, and tour guide services have become an indispensable part of the tourism industry. Traditional tour guide services are usually provided by on-site tour guides, who introduce attractions, history and culture to tourists through verbal explanations, instructions and exhibits. However, there are some limitations to traditional tour guide services, such as the tour guide's knowledge level and verbal expression ability may be limited, the communication between the tour guide and tourists may be affected by language and cultural differences, and the tour guide may not be able to meet the needs of all tourists at busy attractions. In order to solve these problems, various technology-based tour guide service solutions have emerged in recent years, including tour guide apps, virtual tour guide systems, augmented reality (AR) tour guides, etc. These technical solutions utilize technologies such as mobile Internet, smart phones, virtual reality (VR), augmented reality (AR), etc. to provide tourists with a more convenient, personalized and rich tour guide experience.

[0003] Although the existing technology has made some progress in improving the tour guide service experience, there are still some limitations and challenges. First, most tour guide apps and virtual tour guide systems still rely on users to find information and select attractions on their own, lacking personalized and customized tour guide route planning. Secondly, the introduction and explanation of attractions in the existing technology are usually presented in the form of text, pictures or simple videos, lacking immersive and interactive experience, and it is difficult to attract users' attention and enhance their sense of participation. Furthermore, the positioning accuracy and navigation accuracy of some tour guide systems are limited, resulting in users encountering problems such as positioning deviation and unclear routes during use, which affects the user experience and the quality of tour guide services. In addition, although augmented reality (AR) technology has brought new possibilities for tour guide services, there are still some challenges in its practical application. For example, the AR tour guide system has high requirements for equipment performance and stability, requires powerful computing power and high-precision sensors, and may be affected by factors such as light and obstructions in outdoor environments, resulting in poor display effects. In addition, the content production and update costs of the AR tour guide system are high, requiring a professional technical team and rich content resource support, which limits its promotion and popularity in the market. Summary of the invention

[0004] The main purpose of the present invention is to provide a virtual reality tour guide system, which has the advantages of high accuracy and high efficiency.

[0005] In order to solve the above problems, the technical solution of the present invention is achieved as follows:

[0006] A virtual reality tour guide system, the system comprising: an image acquisition unit, used for capturing a panoramic image of an environment on a tour guide route to obtain a panoramic image sequence; preprocessing the panoramic image sequence to obtain a preprocessed panoramic image sequence; a feature extraction unit, used for extracting feature points from the preprocessed panoramic image sequence to obtain a feature point set; a path planning unit, used for planning a tour guide route according to tour guide route information input by a user to obtain a key point set on the tour guide route; a feature matching unit, used for matching the feature point set with the key point set to obtain a matching pair; a posture estimation unit, used for estimating a camera posture according to the matching pair; a virtual model loading unit, used for loading a virtual model according to the key point set to obtain a virtual model set; a scenic spot rendering unit, used for rendering a scenic spot according to the camera posture and the virtual model set to obtain a rendered scenic spot image; and a user experience unit, used for presenting the rendered scenic spot image to a user through an immersive display device, so that the user experiences the tour guide process through the immersive display device.

[0007] Furthermore, the image acquisition unit is a panoramic camera, which captures environmental images on the tour guide route to obtain a panoramic image sequence. ,in represents time; the process of preprocessing the panoramic image sequence includes: denoising and color correction, and the preprocessed panoramic image sequence is .

[0008] Furthermore, the feature extraction unit extracts feature points from the preprocessed panoramic image sequence to obtain a feature point set, which includes: Each frame of image corresponding to each time in the image is connected in chronological order to form an overall panoramic image; for each pixel point in the overall panoramic image , the local structure tensor is calculated using the following formula :

[0009] ;

[0010] in, Indicated in pixels The neighborhood of the center, is the weight function, and Represents pixel points The horizontal and vertical gradients at . For the local structure tensor Use the QR method to perform eigenvalue decomposition and obtain its eigenvalue and ,in ; Determine the feature points according to the size of the eigenvalue, including: if and If all of them are greater than the set threshold, the pixel is considered as a feature point; if and If all are smaller than the threshold, the pixel is considered as a non-feature point; if Greater than the threshold If it is less than the threshold, the pixel It is regarded as an edge point and is not processed. Through the above steps, the feature point set of each frame image is obtained:

[0011] ,

[0012] in Indicates the number of feature points; Is the subscript.

[0013] Furthermore, the navigation information input by the user includes: a starting point, an end point and a must-go point; the path planning unit, according to the guide route information input by the user, plans the guide route in a process including: connecting the starting point, the end point and the must-go point through a road connecting line to obtain the guide route; taking the starting point, the end point and the must-go point on the guide route as key points, and at the same time, randomly selecting multiple points at equal intervals in addition to the starting point, the end point and the must-go point according to the set distance value as key points, and setting the distance value. The following constraints are satisfied:

[0014] ;

[0015] in, is the total length of the guided tour route; The number of points selected at random equal intervals; It is the total number of starting points, end points and must-go points.

[0016] Furthermore, the feature matching unit matches the feature point set with the key point set to obtain a matching pair, including: the image acquisition unit captures a panoramic image of the environment on the tour guide route to obtain a panoramic image sequence; therefore, there is a corresponding relationship between the positions of the tour guide route and the preprocessed panoramic image sequence, and through this corresponding relationship, the corresponding position of each feature point in the tour guide route is found; the distance between the corresponding position of each feature point in the tour guide route and each key point is calculated, and the feature point closest to each key point is matched with the key point to obtain a matching pair.

[0017] Furthermore, the posture estimation unit estimates the camera posture according to the matching pair, and the method includes: assuming that the key point set is:

[0018] ,

[0019] As the subscript, feature point matching is performed, and the matching pair obtained is ; Get the camera intrinsic matrix ; For each matching pair , based on the current estimated camera pose , by projecting the three-dimensional point in the world coordinate system Convert to a 2D point in the camera coordinate system ; Then, calculate the feature points With the projection point The goal is to optimize the camera pose by minimizing the reprojection error of all matching pairs. , set the objective function as:

[0020] ;

[0021] in, represents the Euclidean distance, represents the projection function, and are the rotation matrix and translation vector of the camera respectively; the camera pose is iteratively adjusted using an optimization algorithm , until the reprojection error is within the set range; finally output the optimized camera pose , which includes the rotation matrix and translation vector .

[0022] Furthermore, the camera pose is iteratively adjusted using an optimization algorithm , until the reprojection error is within the set range. The method includes: initially, giving an initial camera pose ; For each matching pair , based on the current estimated camera pose , use the following formula to calculate the reprojection error:

[0023] ;

[0024] The sum of squared reprojection errors is used as the objective function, namely:

[0025] ;

[0026] The objective function is calculated relative to the camera pose by the finite difference method The Jacobian matrix of the objective function is the gradient of the objective function. According to the Jacobian matrix and the gradient of the objective function, the update step size is calculated using the following formula :

[0027] ;

[0028] in, is the Jacobian matrix; is the reprojection error vector, each element of which is a reprojection error; is the tuning parameter that controls the step size, is the identity matrix; Represents the Frobenius norm of the matrix; represents the second-order L2 norm; is the transpose operation of the matrix; Represents the operation of finding the rank of the matrix; based on the calculated update step size , update the current camera pose :

[0029] ;

[0030] in, is the updated camera pose; determine whether the reprojection error is within the set threshold range, if it is satisfied, stop the iteration, otherwise continue the iteration.

[0031] Furthermore, the virtual model loading unit is based on the key point set , load the virtual model corresponding to the key point; since each key point corresponds to a point on the tour guide route, according to the position of the point in the tour guide route, find the pre-established virtual model corresponding to the position to obtain the virtual model set:

[0032] ;

[0033] in, is the number of virtual models; Is the subscript.

[0034] Furthermore, the scenic spot rendering unit performs scenic spot rendering according to the camera posture and the virtual model set, and the method for obtaining the rendered scenic spot image includes: , the virtual model collection The vertex coordinates of the virtual model are transformed into the camera coordinate system to project each virtual model into the camera's view. For each pixel in the camera's view, a ray is emitted through the scene to detect whether the ray intersects with an object in the scene. If so, the color at the intersection is calculated based on the material properties of the object. The rendering equation is used to calculate the color of each pixel. For the point where the ray intersects with the object, it is determined whether the point is in the shadow. If so, its color is set to the shadow color. If the surface has specular reflection or refraction properties, the reflection or refraction of the light is recursively calculated. The integral term in the rendering equation is numerically integrated or Monte Carlo sampled to calculate the color of each pixel to obtain a rendered scenic spot image. .

[0035] The virtual reality tour guide system of the present invention has the following beneficial effects:

[0036] The posture estimation unit of the present invention estimates the posture of the camera using matching pairs, and estimates the position and orientation of the camera in real time by matching the positions of feature points in the tour guide route with key points, thereby accurately determining the position and orientation of the camera in the tour guide route. This real-time posture estimation can effectively improve the positioning accuracy and navigation accuracy of the tour guide system, making it more convenient and smooth for users to use it.

[0037] The virtual model loading unit of the present invention loads the virtual model according to the key point set, so that each key point corresponds to a point on the tour guide route, and according to the position of the point in the tour guide route, a pre-established corresponding virtual model is selected, thereby achieving accurate correspondence between the scenic spot and the virtual model. This rich virtual model loading can present more realistic and vivid scenic spot images to users, enhancing the user's sense of immersion and experience.

[0038] The scenic spot rendering unit of the present invention performs scenic spot rendering according to the camera posture and the virtual model set, uses ray tracing and rendering equations to simulate the propagation and interaction of light in the scene, calculates the color of each pixel, and thus generates the final rendered scenic spot image. This realistic scenic spot rendering technology can provide users with a high-quality visual experience, making users feel as if they are in a real scenic spot environment, enhancing the user's sense of immersion and participation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the system structure of a virtual reality tour guide system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] Example 1: Reference Figure 1, a virtual reality tour guide system, the system comprising: an image acquisition unit, used to capture a panoramic image of an environment on a tour guide route to obtain a panoramic image sequence; preprocessing the panoramic image sequence to obtain a preprocessed panoramic image sequence; a feature extraction unit, used to extract feature points from the preprocessed panoramic image sequence to obtain a feature point set; a path planning unit, used to plan a tour guide route according to tour guide route information input by a user, to obtain a key point set on the tour guide route; a feature matching unit, used to match the feature point set with the key point set to obtain a matching pair; a posture estimation unit, used to estimate a camera posture according to the matching pair; a virtual model loading unit, used to load a virtual model according to the key point set to obtain a virtual model set; a scenic spot rendering unit, used to render a scenic spot according to the camera posture and the virtual model set to obtain a rendered scenic spot image; a user experience unit, used to present the rendered scenic spot image to a user through an immersive display device, and the user experiences the tour guide process through the immersive display device.

[0042] Specifically, in the image acquisition unit, a panoramic camera or multiple cameras are usually used to capture environmental images. The panoramic camera has a wide-angle lens that can capture a wider scene, while multiple cameras can achieve panoramic image acquisition through stitching technology. These cameras are usually installed at key locations on the tour guide route to ensure that important attractions and environments during the tour guide process are captured. Once the camera captures the environmental panoramic image, the image acquisition unit transmits it to the subsequent processing module of the system. During the transmission process, a compression algorithm may be used to compress the image to reduce the bandwidth requirement for data transmission. At the same time, in order to improve the stability and reliability of the system, the image acquisition unit may also perform real-time monitoring and fault handling to ensure the continuity and integrity of the image data. In addition, the image acquisition unit also needs to consider the impact of environmental conditions on image quality. For example, in the case of insufficient light or strong light reflection, the camera may have problems of underexposure or overexposure, thereby affecting the quality of the captured image. Therefore, the image acquisition unit may be equipped with automatic exposure and white balance adjustment functions to cope with image capture requirements under different environmental conditions.

[0043] The feature extraction unit is usually implemented using algorithms in the field of computer vision, the most common of which include SIFT (Scale Invariant Feature Transform), SURF (Speeded Robust Features) and ORB (Oriented FAST and Rotated BRIEF). The basic principle of these algorithms is to find local feature points in the image and calculate the descriptors of these feature points for subsequent matching and recognition. In the process of feature extraction, the algorithm usually detects a series of possible key points in the image first, which are usually corners, edges or spots in the image. Then, for each key point, the algorithm calculates the feature descriptor of the surrounding area. The descriptor is usually a vector that describes the texture, color and gradient features of the area around the key point. The key to the feature extraction algorithm is how to select appropriate key points and calculate stable and discriminative descriptors. This involves how to find local extreme points in the image and design feature descriptions for them. For example, the SIFT algorithm detects local extreme points in the image and constructs descriptors based on the gradient information around these extreme points to achieve scale invariance and rotation invariance. Compared with traditional feature extraction methods, current algorithms usually have higher computational efficiency and better robustness. For example, the ORB algorithm combines the FAST and BRIEF algorithms, providing feature description performance comparable to SIFT and SURF while maintaining computational speed. In short, the feature extraction unit extracts stable and distinguishing feature points from the preprocessed panoramic image sequence through an algorithm, providing an important data basis for subsequent guide route planning, feature matching, and posture estimation modules. The key lies in how to select appropriate feature points and design effective feature descriptors to achieve accurate recognition and matching of different scenes.

[0044] The feature matching unit is usually implemented using feature matching algorithms in the field of computer vision, among which the most common ones include nearest neighbor matching, RANSAC algorithm and optical flow method. The basic principle of these algorithms is to find similar feature points in two images and match them to establish a correspondence between the two images. In the process of feature matching, it is first necessary to extract a set of feature points from the two images and calculate the descriptor of each feature point. Then, using the feature matching algorithm, find the corresponding feature points between the two feature point sets, that is, find the feature point pairs representing the same actual scenic spots in the two images. One of the simplest feature matching algorithms is the nearest neighbor matching, which calculates the distance between the feature points in the two images and selects the closest feature points for matching. However, due to factors such as noise and occlusion in the image, the nearest neighbor matching may lead to the problem of mismatching, so it is usually necessary to combine other algorithms for optimization. The RANSAC algorithm is a robust feature matching algorithm, which selects the inliers that meet the model from the feature point set by random sampling and model fitting, thereby eliminating the influence of noise and mismatching and obtaining more accurate matching results. Optical flow is a pixel-level feature matching algorithm that matches feature points by tracking the motion trajectory of pixels in an image. The key to the feature matching unit is how to select a suitable feature matching algorithm and combine image processing technology with geometric principles to achieve accurate matching of feature points. Only by establishing the correspondence between feature points in an image and actual scenic spots can we ensure that subsequent modules such as posture estimation and virtual model loading can correctly locate and render the virtual model.

[0045] The pose estimation unit is usually implemented using pose estimation algorithms in the field of computer vision, the most common of which include PnP problem solving based on feature point matching, SLAM algorithm based on structured light, and sensor-based inertial measurement unit (IMU) fusion. In the process of pose estimation, it is first necessary to obtain matching pairs through the feature matching unit, that is, the correspondence between feature points in the image and key points in the actual scene. Then, using these matching pairs, the pose estimation unit can calculate the pose of the camera by solving the PnP problem based on the camera's intrinsic and extrinsic parameters. The PnP problem is a classic computer vision problem, and its goal is to estimate the position and pose of the camera based on known 3D space points and their projection points in the image. Common PnP problem solving algorithms include EPnP algorithm, DLS algorithm, and P3P algorithm. These algorithms obtain the pose of the camera by minimizing the reprojection error or solving a set of polynomial equations. In addition to the PnP algorithm based on feature point matching, the SLAM algorithm is also a commonly used pose estimation method, which estimates the position and pose of the camera in real time by continuously fusing the visual information from the camera and the inertial information from the sensor (such as IMU). SLAM algorithms can be divided into feature-based SLAM and direct-based SLAM. Feature-based SLAM achieves attitude estimation by extracting feature points in the image, while direct-based SLAM directly uses pixel information in the image to estimate the motion of the camera. In summary, the attitude estimation unit estimates the position and attitude of the camera based on matching pairs by solving the PnP problem or using the SLAM algorithm to achieve accurate positioning and rendering of the virtual model. The key lies in selecting a suitable attitude estimation algorithm and using image features and sensor information to estimate the camera's position and attitude in real time, thereby improving the stability and accuracy of the virtual tour guide system.

[0046] The virtual model loading unit is usually implemented using computer graphics technology and a three-dimensional model library. These tools and libraries provide a wealth of virtual model resources and rendering engines, which can achieve high-quality virtual environment reconstruction and rendering effects. In the process of virtual model loading, it is first necessary to determine the virtual model to be loaded based on the key point set. The key point set is usually provided by the feature extraction and path planning unit, which contains the scenic spots or key locations of interest to the user. Then, the virtual model loading unit selects a suitable model from the virtual model library based on these key points and loads it into the virtual environment. The virtual model loading unit is responsible for matching the selected virtual model with the actual environment and accurately positioning and placing it. This involves how to place the virtual model in the correct position according to the position and posture of the key points, and adjust its size and orientation to match the actual environment. Generally, technologies such as three-dimensional transformation and affine transformation can be used to achieve the positioning and placement of the virtual model. The virtual model loading unit also needs to consider the interaction and consistency between the virtual model and the actual environment. For example, the appearance and material of the virtual model should match the actual environment to ensure that the user gets a sense of reality and immersion in the virtual environment. In addition, the performance and rendering efficiency of the virtual model need to be considered to ensure the smoothness and stability of the system. In short, the virtual model loading unit uses computer graphics technology and a 3D model library to load the corresponding virtual model according to the key point set, and places it in the correct position and posture to achieve virtual reconstruction and rendering of the actual environment. The key lies in selecting appropriate virtual model resources and rendering engines, and achieving accurate positioning and consistent matching of virtual models, thereby providing users with a high-quality virtual tour guide experience.

[0047] The scenic spot rendering unit is usually implemented using computer graphics technology and rendering engines, the most common of which include OpenGL, DirectX and Vulkan. These rendering engines provide rich rendering functions and effects, and can achieve high-quality virtual scene rendering and real-time interaction. In the process of scenic spot rendering, it is first necessary to build a scene graph based on the camera pose and virtual model set, that is, to place the virtual model to be rendered within the camera's field of view and calculate its position projected onto the image plane. Then, the scenic spot rendering unit uses the rendering engine to render the scene graph to generate the final scenic spot image.

[0048] Embodiment 2: The image acquisition unit is a panoramic camera, which captures environmental images along the tour guide route to obtain a panoramic image sequence ,in represents time; the process of preprocessing the panoramic image sequence includes: denoising and color correction, and the preprocessed panoramic image sequence is .

[0049] Specifically, a panoramic camera is a special camera device that is usually equipped with multiple lenses and can capture 360-degree images around it at the same time. Each frame of the sequence contains an image of the environment from a panoramic perspective, which can be provided to subsequent processing units. The main purpose of the preprocessing process is to optimize the captured panoramic image sequence to improve the quality and accuracy of the image. Color correction is one of the important operations, which is used to adjust the color and brightness of the image to make it more realistic and clear. Other preprocessing operations may also include removing lens distortion, noise reduction, etc. to further improve image quality. Preprocessed panoramic image sequence While retaining the panoramic perspective of the original image, it has been processed by color correction to make the image more realistic and clear. Such preprocessing can improve the accuracy and efficiency of subsequent processing units, such as feature extraction, path planning, feature matching and other modules. The process of preprocessing the panoramic image sequence includes: Each frame in is divided into blocks, and the size is A collection of overlapping blocks:

[0050] ,

[0051] in Represents the number of blocks; two blocks are calculated using the following formula and Similarities between :

[0052] ;

[0053] in, Representation Block Medium Pixels The gray value of Parameters for controlling similarity; represents the second-order L2 norm; the block is calculated using the following formula Weight :

[0054] ;

[0055] The calculated weights are used to perform weighted averaging on each pixel to obtain a denoised panoramic image sequence. ; The denoised panoramic image sequence Convert to LAB color space; for each channel in LAB color space, use the following formula to perform polynomial fitting:

[0056] ;

[0057] ;

[0058] ;

[0059] in, , , represents the corrected LAB color space channel, , , represents the original LAB color space channels, , , is the fitting coefficient; finally, the corrected LAB color space is converted back to the RGB color space to obtain the preprocessed panoramic image sequence .

[0060] Embodiment 3: A feature extraction unit extracts feature points from a pre-processed panoramic image sequence. The method for obtaining a feature point set includes: Each frame of image corresponding to each time in the image is connected in chronological order to form an overall panoramic image; for each pixel point in the overall panoramic image , the local structure tensor is calculated using the following formula :

[0061] ;

[0062] in, Indicated in pixels The neighborhood of the center, is the weight function, and Represents pixel points The horizontal and vertical gradients at . For the local structure tensor Use the QR method to perform eigenvalue decomposition and obtain its eigenvalue and ,in ; Determine the feature points according to the size of the eigenvalue, including: if and If all of them are greater than the set threshold, the pixel is considered as a feature point; if and If all are smaller than the threshold, the pixel is considered as a non-feature point; if Greater than the threshold If it is less than the threshold, the pixel It is regarded as an edge point and is not processed. Through the above steps, the feature point set of each frame image is obtained:

[0063] ,

[0064] in Indicates the number of feature points; Is the subscript.

[0065] Specifically, first, the panoramic image sequence is connected in chronological order to form an overall panoramic image, and then the local structure tensor calculation method is used to find the feature points in the image. It is a matrix that describes the local features of an image and is used to reflect the image at a certain point. The process of calculating the local structure tensor is to calculate the local structure tensor by The gradient information of pixels in the neighborhood of is weighted and accumulated, and a second-order symmetric matrix is ​​constructed. In the calculation process of the local structure tensor, the pixel point is first calculated. Horizontal and vertical gradients and Then, using this gradient information and the weight function Perform weighted accumulation of pixels in the neighborhood to obtain the local structure tensor In this process, the weight function is used to measure the distance and similarity between different pixels to ensure that the local structure tensor can accurately reflect the local structure information of the image. Local Structure Tensor Perform eigenvalue decomposition to obtain its eigenvalue and Eigenvalue decomposition is a method of decomposing a matrix into eigenvectors and eigenvalues, where the eigenvalues ​​reflect the degree of change of the matrix in a specific direction, and the eigenvectors represent these directions. Here, the eigenvalues ​​are arranged in order from large to small. Indicates the degree of change in the main direction. It indicates the degree of change in the secondary direction. Finally, the feature points in the image are determined according to the size of the eigenvalue. If both the primary and secondary eigenvalues ​​are greater than the set threshold, it means that there is an obvious structural change around the point, so it is regarded as a feature point; if both eigenvalues ​​are less than the threshold, it means that the structural change around the point is not obvious, and it is regarded as a non-feature point; if the primary eigenvalue is greater than the threshold and the secondary eigenvalue is less than the threshold, it means that the point may be located at the edge and will not be processed.

[0066] The feature extraction unit extracts key feature points from the preprocessed panoramic image sequence by calculating the local structure tensor and eigenvalue decomposition method. These feature points have significant local structure information and can accurately describe important features and areas in the image, such as prominent scenic spots, building edges, etc. Therefore, the role of feature extraction is to efficiently abstract and describe the image, providing an important basis for subsequent image processing and analysis. Secondly, the extraction of feature points provides key data support for modules such as path planning and feature matching in the virtual tour guide system. The path planning module can use the extracted feature point set to plan the tour route according to the user's input information and determine the optimal scenic spot tour path. The feature matching module can use the feature point set to match and locate between images, realize accurate loading and rendering of virtual models, and provide users with a more realistic and immersive tour guide experience. In addition, the feature extraction unit can also be used for scene reconstruction and recognition. By modeling and analyzing the extracted feature points, three-dimensional reconstruction and recognition of the scene can be realized, providing the virtual tour guide system with richer and more accurate scene information. These scene information can be used for the augmented reality functions of the virtual tour guide system, such as real-time scenic spot recognition, location annotation, etc., to enhance the user's tour guide experience and interactivity.

[0067] Embodiment 4: The navigation information input by the user includes: a starting point, an end point and a must-go point; the path planning unit, according to the guide route information input by the user, plans the guide route in a process including: connecting the starting point, the end point and the must-go point through a road connecting line to obtain the guide route; taking the starting point, the end point and the must-go point on the guide route as key points, and at the same time, randomly selecting multiple points at equal intervals in addition to the starting point, the end point and the must-go point according to the set distance value as key points, and setting the distance value The following constraints are satisfied:

[0068] ;

[0069] in, is the total length of the guided tour route; The number of points selected at random equal intervals; It is the total number of starting points, end points and must-go points.

[0070] Specifically, the distance value Certain constraints need to be satisfied, namely The value range of needs to be within a reasonable range. The upper and lower limits of this range are determined by the total length of the guided route. , the number of randomly selected points and the number of starting points, end points, and required points This constraint relationship ensures that the selected additional key points can be evenly distributed on the guide route and will not be too dense or too sparse.

[0071] Embodiment 5: A feature matching unit matches a feature point set with a key point set to obtain a matching pair, comprising: an image acquisition unit captures a panoramic image of the environment on a tour guide route to obtain a panoramic image sequence; therefore, there is a corresponding relationship between the positions of the tour guide route and the preprocessed panoramic image sequence, and through this corresponding relationship, the corresponding position of each feature point in the tour guide route is found; the distance between the corresponding position of each feature point in the tour guide route and each key point is calculated, and the feature point closest to each key point is matched with the key point to obtain a matching pair.

[0072] Specifically, since the image acquisition unit captures the panoramic image sequence on the tour guide route, each position in the tour guide route can correspond to a specific frame image in the preprocessed panoramic image sequence. The feature matching unit uses this correspondence to find the corresponding position of each feature point in the tour guide route, thereby associating the feature point with the tour guide route. Next, the feature matching unit calculates the distance between the corresponding position of each feature point in the tour guide route and each key point. The key points here include the starting point, the end point and the must-pass point on the tour guide route, as well as the additional key points selected according to the method in Example 4. By calculating the distance, the feature matching unit can find the feature point closest to each key point, and then match these feature points with the corresponding key points. Finally, the feature matching unit obtains the matching pairs between the feature points and the key points, and establishes the corresponding relationship between them. These matching pairs will be used as input data for subsequent posture estimation, virtual model loading and other modules, and are used to accurately locate and load virtual models, so as to achieve accurate description and reconstruction of the tour guide route and scene.

[0073] Embodiment 6: A method for estimating a camera posture according to a matching pair by a posture estimation unit includes: assuming that a key point set is:

[0074] ,

[0075] As the subscript, feature point matching is performed, and the matching pair obtained is ; Get the camera intrinsic matrix ; For each matching pair , based on the current estimated camera pose , by projecting the three-dimensional point in the world coordinate system Convert to a 2D point in the camera coordinate system ; Then, calculate the feature points With the projection point The goal is to optimize the camera pose by minimizing the reprojection error of all matching pairs. , set the objective function as:

[0076] ;

[0077] in, represents the Euclidean distance, represents the projection function, and are the rotation matrix and translation vector of the camera respectively; the camera pose is iteratively adjusted using an optimization algorithm , until the reprojection error is within the set range; finally output the optimized camera pose , which includes the rotation matrix and translation vector .

[0078] Specifically, the attitude estimation unit obtains the camera's intrinsic parameter matrix , which describes the internal parameters of the camera, such as focal length, principal point, etc. Intrinsic parameter matrix is a It can project the 3D point in the world coordinate system to the 2D point in the camera coordinate system, i.e., the pixel coordinate. ,in is a feature point, is the key point. The posture estimation unit needs to estimate the current camera posture according to the current estimated , to make the key points Convert to a one-dimensional point in the camera coordinate system This process involves the camera's external parameters, namely the camera's rotation matrix and translation vector By placing the key points Multiply by the camera's rotation matrix , and add the camera's translation vector , you can put the key points Convert to the camera coordinate system to get the projection point Next, the posture estimation unit needs to calculate the feature points With the projection point The distance between them is used as the reprojection error. The reprojection error represents the difference between the observed position of the feature point in the image and the projected position obtained by camera pose estimation. The Euclidean distance is used to measure this difference, that is, to calculate the feature point and the projection point Ultimately, the goal of the pose estimation unit is to optimize the camera pose by minimizing the reprojection error of all matching pairs. To this end, an objective function is defined that contains the sum of the squares of the reprojection errors of all matching pairs. The pose estimation unit uses an optimization algorithm to iteratively adjust the camera pose , which is the rotation matrix of the camera and translation vector , so that the objective function reaches the minimum value. By continuously adjusting the camera pose , the reprojection error will gradually decrease until it reaches the set range.

[0079] Example 7: Iteratively adjust the camera pose using an optimization algorithm , until the reprojection error is within the set range. The method includes: initially, giving an initial camera pose ; For each matching pair , based on the current estimated camera pose , use the following formula to calculate the reprojection error:

[0080] ;

[0081] The sum of squared reprojection errors is used as the objective function, namely:

[0082] ;

[0083] The objective function is calculated relative to the camera pose by the finite difference method The Jacobian matrix of the objective function is the gradient of the objective function. According to the Jacobian matrix and the gradient of the objective function, the update step size is calculated using the following formula :

[0084] ;

[0085] in, is the Jacobian matrix; is the reprojection error vector, each element of which is a reprojection error; is the tuning parameter that controls the step size, is the identity matrix; Represents the Frobenius norm of the matrix; represents the second-order L2 norm; is the transpose operation of the matrix; Represents the operation of finding the rank of the matrix; based on the calculated update step size , update the current camera pose :

[0086] ;

[0087] in, is the updated camera pose; determine whether the reprojection error is within the set threshold range, if it is satisfied, stop the iteration, otherwise continue the iteration.

[0088] Specifically, the right side of the formula is a vector ,in is the Jacobian matrix, is the reprojection error vector, which indicates the gradient direction of the objective function. This vector represents the objective function at the current camera pose. Next, the left side of the formula is a matrix equation, which contains a coefficient matrix and , indicating the direction and size of the update step. This coefficient matrix is ​​composed of the Jacobian matrix Specifically: The term represents the inner product of the Jacobian matrix, which is used to measure the directionality and size of the Jacobian matrix. This term makes the update step size Able to move along the gradient direction of the objective function to reduce the value of the objective function as quickly as possible. It represents the Frobenius norm of the Jacobian matrix, which is used to measure the overall size of the Jacobian matrix. This item can control the overall size of the update step to avoid the instability of the optimization process caused by too large an update step. It represents the square of the second-order L2 norm of the Jacobian matrix, and is also used to control the size of the update step. This term can avoid unreasonable update step sizes when the matrix is ​​too sparse or too dense. represents a control parameter, where is the parameter that controls the step size adjustment, is the rank of the Jacobian matrix. This term can adjust the step size according to the rank of the Jacobian matrix to make the update step size more reasonable. Taking all these factors into consideration, the left side of the formula can be regarded as an adjusted step size update equation, which calculates the appropriate update step size by considering factors such as the direction, size, and sparsity of the Jacobian matrix. Finally, according to the calculated update step size , you can update the current camera pose , and then continue to optimize the objective function until the reprojection error is within the set range. This process realizes the iterative adjustment of the camera pose, so that the reprojection error gradually decreases and finally reaches the set threshold range.

[0089] Embodiment 8: Virtual model loading unit according to key point set , load the virtual model corresponding to the key point; since each key point corresponds to a point on the guide route, according to the position of the point in the guide route, find the pre-established virtual model corresponding to the position, and obtain the virtual model set ;in, is the number of virtual models; Is the subscript.

[0090] Specifically, the virtual model loading unit first obtains the key point set , these key points are important locations on the user's guide route. For each key point , the virtual model loading unit finds the pre-established virtual model corresponding to the position of the key point in the guide route. These virtual models can be pre-designed three-dimensional models, representing scenic spots, buildings or other objects in the real world. The process of loading virtual models can be achieved by reading model files, database queries or other methods. After loading the virtual models corresponding to all key points, the virtual model loading unit obtains a virtual model set:

[0091] ,

[0092] in is the number of virtual models, Indicates This virtual model collection contains all the virtual models at the corresponding key points on the tour guide route, which are used for subsequent scenic spot rendering and user experience.

[0093] Embodiment 9: A scenic spot rendering unit performs scenic spot rendering according to a camera posture and a virtual model set, and a method for obtaining a rendered scenic spot image comprises: , the virtual model collection The vertex coordinates of the virtual model are transformed into the camera coordinate system to project each virtual model into the camera's view. For each pixel in the camera's view, a ray is emitted through the scene to detect whether the ray intersects with an object in the scene. If so, the color at the intersection is calculated based on the material properties of the object. The rendering equation is used to calculate the color of each pixel. For the point where the ray intersects with the object, it is determined whether the point is in the shadow. If so, its color is set to the shadow color. If the surface has specular reflection or refraction properties, the reflection or refraction of the light is recursively calculated. The integral term in the rendering equation is numerically integrated or Monte Carlo sampled to calculate the color of each pixel to obtain a rendered scenic spot image. .

[0094] Specifically, ray tracing is one of the core technologies in Example 9. For each pixel in the camera's perspective, a ray is emitted through the scene to detect whether the ray intersects with the virtual model in the scene. This is because the propagation and interaction of light determine the formation of the final image during the rendering process. Through ray tracing, the propagation process of light in the scene can be simulated to determine the final color of each pixel. When performing ray tracing, the material properties of the object need to be considered. Objects of different materials will show different reflection, refraction and absorption characteristics under the action of light, so it is necessary to calculate the color at the intersection according to the material properties of the object. This is done to improve the realism and fidelity of the rendering, so that users can better feel the authenticity and details of the scene. The rendering equation is a mathematical model that describes the propagation and interaction of light in the scene. The equation takes into account the interaction between light and objects in the scene, including diffuse reflection, specular reflection, refraction, etc. By solving the rendering equation, the final color of each pixel can be calculated, thereby generating the final rendered scenic spot image. The application of the rendering equation is to consider the various effects of light in the rendering process, so as to more accurately simulate the lighting effects in the real world. In addition, the calculation of shadows also needs to be considered. Shadows are caused by the fact that some areas do not receive light due to the light being blocked by objects, so it is necessary to determine whether the intersection is in the shadow and adjust its color accordingly. This is to enhance the realism and fidelity of the rendered image, so that users can better feel the lighting effects and three-dimensional sense in the scene. Finally, through the numerical integration or Monte Carlo sampling of the rendering equation, the color of each pixel can be calculated to obtain the rendered scenic spot image. This image contains the visual presentation of all virtual models in the scene, and is the scenic spot image that the user finally observes, providing users with an immersive and realistic tour guide experience.

[0095] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual reality tour guide system, characterized in that: The system comprises: an image acquisition unit, which is used to capture a panoramic image of the environment on the tour guide route to obtain a panoramic image sequence; preprocessing the panoramic image sequence to obtain a preprocessed panoramic image sequence; a feature extraction unit, which is used to extract feature points from the preprocessed panoramic image sequence to obtain a feature point set; a path planning unit, which is used to plan the tour guide route according to the tour guide route information input by the user to obtain a key point set on the tour guide route; a feature matching unit, which is used to match the feature point set with the key point set to obtain a matching pair; a posture estimation unit, which is used to estimate the camera posture according to the matching pair; a virtual model loading unit, which is used to load the virtual model according to the key point set to obtain a virtual model set; a scenic spot rendering unit, which is used to render the scenic spot according to the camera posture and the virtual model set to obtain a rendered scenic spot image; a user experience unit, which is used to present the rendered scenic spot image to the user through an immersive display device, so that the user experiences the tour guide process through the immersive display device; The method for estimating the camera posture according to the matching pair by the posture estimation unit includes: assuming that the key point set is: j is the subscript, and the feature point matching is performed, and the matching pair obtained is (f i , p j ); Get the camera internal parameter matrix K; For each matching pair (f i , p j ), according to the current estimated camera pose T c , by projecting the three-dimensional point p in the world coordinate system j Transformed into a two-dimensional point p in the camera coordinate system proj ; Then, calculate the feature point f i With the projection point p proj The distance between them is taken as the reprojection error; the goal is to optimize the camera pose T by minimizing the reprojection error of all matching pairs. c , set the objective function as: Where ||·|| represents the Euclidean distance, Π represents the projection function, R and t are the rotation matrix and translation vector of the camera respectively; the camera pose T is iteratively adjusted using the optimization algorithm c , until the reprojection error is within the set range; finally output the optimized camera pose T c , which includes the rotation matrix R and the translation vector t; Use an optimization algorithm to iteratively adjust the camera pose T c , until the reprojection error is within the set range. The method includes: initially, giving an initial camera pose T c ; For each matching pair (f i , p j ), according to the current estimated camera pose T c , use the following formula to calculate the reprojection error: e i =||f i -Π(Rp j +t)||; The sum of squared reprojection errors is used as the objective function, namely: The objective function is calculated by the finite difference method relative to the camera pose T c The Jacobian matrix of the objective function is the gradient of the objective function. According to the Jacobian matrix and the gradient of the objective function, the update step length δT is calculated using the following formula c : Where J is the Jacobian matrix; e is the reprojection error vector, each element of which is a reprojection error; λ is the adjustment parameter that controls the step size, and I is the identity matrix; |||| F Represents the Frobenius norm of the matrix; represents the second-order L2 norm; T is the transpose operation of the matrix; rank() represents the operation of finding the rank of the matrix; the update step length δT is calculated according to c , update the current camera pose T c : T c，new ←T c +δT c ; Among them, T c，new is the updated camera pose; determine whether the reprojection error is within the set threshold range, if it is satisfied, stop the iteration, otherwise continue the iteration.

2. The virtual reality tour guide system according to claim 1, characterized in that: The image acquisition unit is a panoramic camera, which captures the environment image on the tour guide route to obtain a panoramic image sequence I pano (t), where t represents time; The process of preprocessing the panoramic image sequence includes: denoising and color correction. The preprocessed panoramic image sequence is 3. The virtual reality tour guide system according to claim 2, characterized in that: The feature extraction unit extracts feature points from the pre-processed panoramic image sequence, and the method for obtaining the feature point set includes: Each frame of image corresponding to each time in the image is connected end to end in chronological order to form an overall panoramic image; for each pixel point (x, y) in the overall panoramic image, the local structure tensor S(x, y) is calculated using the following formula: Among them, N(x, y) represents the neighborhood centered on the pixel point (x, y), w(x, y, p, q) is the weight function, and I x (p, q) and I y (p, q) represent the horizontal gradient and vertical gradient at the pixel point (p, q) respectively. The local structure tensor S(x, y) is decomposed by the QR method to obtain its eigenvalues ​​λ1 and λ2, where λ1≥λ2; the feature points are determined according to the size of the eigenvalues, specifically including: if λ1 and λ2 are both greater than the set threshold, the pixel point (x, y) is regarded as a feature point; if λ1 and λ2 are both less than the threshold, the pixel point (x, y) is regarded as a non-feature point; if λ1 is greater than the threshold and λ2 is less than the threshold, the pixel point (x, y) is regarded as an edge point and is not processed; through the above steps, the feature point set of each frame image is obtained: Where N represents the number of feature points; i is a subscript.

4. The virtual reality tour guide system according to claim 3, characterized in that: The navigation information input by the user includes: a starting point, an end point and a must-go point; the path planning unit, according to the guide route information input by the user, plans the guide route in the process of: connecting the starting point, the end point and the must-go point through a road connecting line to obtain the guide route; taking the starting point, the end point and the must-go point on the guide route as key points, and at the same time, randomly selecting multiple points at equal intervals in addition to the starting point, the end point and the must-go point according to the set distance value as key points, and setting the distance value D to satisfy the following constraint relationship: Among them, DIS is the total length of the guide route; M is the number of randomly selected points with equal spacing; Q is the total number of starting points, end points and must-pass points.

5. The virtual reality tour guide system according to claim 4, characterized in that: The feature matching unit matches the feature point set with the key point set to obtain a matching pair, which includes: the image acquisition unit captures a panoramic image of the environment on the tour guide route to obtain a panoramic image sequence; therefore, there is a corresponding relationship between the positions of the tour guide route and the pre-processed panoramic image sequence, and through this corresponding relationship, the corresponding position of each feature point in the tour guide route is found; the distance between the corresponding position of each feature point in the tour guide route and each key point is calculated, and the feature point closest to each key point is matched with the key point to obtain a matching pair.

6. The virtual reality tour guide system according to claim 1, characterized in that: The virtual model loading unit loads the virtual models corresponding to the key points according to the key point set P; since each key point corresponds to a point on the tour guide route, the virtual model corresponding to the position established in advance at the position is found according to the position of the point in the tour guide route to obtain the virtual model set: Where K is the number of virtual models and k is a subscript.

7. The virtual reality tour guide system according to claim 6, characterized in that: The scenic spot rendering unit performs scenic spot rendering according to the camera posture and the virtual model set, and the method for obtaining the rendered scenic spot image includes: c , the virtual model set M virtual The vertex coordinates of the virtual model are transformed into the camera coordinate system to project each virtual model into the camera's view. For each pixel in the camera's view, a ray is emitted through the scene to detect whether the ray intersects with an object in the scene. If so, the color at the intersection is calculated based on the material properties of the object. The color of each pixel is calculated using the rendering equation. For the point where the ray intersects with the object, it is determined whether the point is in the shadow. If so, its color is set to the shadow color. If the surface has a specular reflection property or a refraction property, the reflection or refraction of the light is recursively calculated. The integral term in the rendering equation is numerically integrated or Monte Carlo sampled to calculate the color of each pixel to obtain the rendered scenic spot image I. render .

Images