A real walking roaming system and method based on panoramic video

Abstract

The present invention provides a real walking roaming system and method based on panoramic video. The present invention can organize the loading order of panoramic videos, simulate and generate large-scale virtual scenes, and support users wearing virtual reality devices to perform immersive virtual roaming in a limited space through real walking interactive roaming, ensuring the user's sense of presence.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual roaming, and in particular relates to a real walking roaming system and method based on panoramic video. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Panoramic video, as an emerging media presentation format, offers a high degree of freedom and immersion. With the introduction of consumer-grade panoramic cameras, it's now possible for people to roam virtual scenes at home using virtual reality devices. However, due to spatial limitations, it's difficult to interactively roam large scenes captured in panoramic video.

[0004] First, most virtual scene navigation methods are categorized as either static or mobile, depending on whether the user moves within the actual space. Static navigation primarily relies on inputting movement commands via devices like controllers or keyboards, which lacks immersion and realism. Mobile navigation offers a more realistic and comfortable experience, but limited physical space makes it difficult for users to navigate large, panoramic video virtual scenes.

[0005] Secondly, most virtual scenes simulate real-world scenes through 3D reconstruction technology. However, this method requires high hardware equipment requirements, has a long production cycle, and requires a large development workload. Furthermore, it is not possible for ordinary users to participate in the creation. Panorama-based content generation involves taking a panoramic image captured at a fixed location, transforming it through stereoscopic projection, and mapping it onto a cube or sphere for display. While this method is simple to operate and supports diverse creative entities, users are limited to viewing the location where the panorama was captured, and continuous roaming is not possible. Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes a real walking roaming system and method based on panoramic video. The present invention can organize the loading order of panoramic videos, simulate and generate large-scale virtual scenes, and support users wearing virtual reality devices to perform immersive virtual roaming in a limited space through real walking interactive roaming, ensuring the user's sense of presence.

[0007] According to some embodiments, the present invention adopts the following technical solutions:

[0008] A real walking roaming method based on panoramic video includes the following steps:

[0009] Obtain a panoramic video of the area to be roamed, where each panoramic video constitutes an independent path sequence;

[0010] Recording the location information of each node and the start and end points and length of each panoramic video segment, generating a two-dimensional map, segmenting the path of the panoramic video based on the nodes, and storing the panoramic video sequence based on the path segments;

[0011] The frame sequence of the panoramic video is located through the mapping relationship between the two-dimensional map position and the number of panoramic video frames, as well as the mapping relationship between the user's physical space information and the position in the virtual space, and the frame selection for continuous frame playback is performed according to the frame sequence.

[0012] As an optional implementation, a method for obtaining a panoramic video of the area to be roamed includes a user creating and shooting a panoramic video by himself or retrieving a panoramic video of an existing scene.

[0013] As an optional implementation, an undirected weighted graph is generated based on the information of each node, and the connection information between the nodes is saved through the adjacency matrix. All values ​​in the adjacency matrix are initialized to zero, and the node serial number is used to find out whether the path exists. If it exists, the corresponding element in the adjacency matrix is ​​updated to store the length of the path. If it does not exist, the corresponding element remains zero.

[0014] As an optional implementation, the mapping relationship between the two-dimensional map position and the panoramic video frame number is to determine the starting node and the ending node of each path, generate a path between the two nodes, and connect the two nodes on a two-dimensional plane, which is represented as a two-dimensional path between the nodes.

[0015] As an optional implementation, it also includes predicting the user's roaming path based on the user's perspective. The specific process includes: when the user arrives at a node, the path is switched according to the user's perspective. The basis for switching is the angle between the user's perspective and each node. Each node records the path information of its adjacent nodes. The angle of each path at the node turning point with the node as the coordinate origin and the forward direction as zero degrees is obtained. The user's perspective direction is compared with the directions of each angle, and the path with the closest angle is selected as the user's next roaming path.

[0016] A real walking and roaming system based on panoramic video, comprising:

[0017] The head-mounted display device interaction module is configured to display the rendered image, determine the user's physical space information, and implement roaming using a redirection algorithm;

[0018] A panoramic video acquisition module is configured to acquire a panoramic video of the area to be roamed, where each panoramic video constitutes an independent path sequence;

[0019] a two-dimensional map module configured to record the location information of each node and the start and end points and length of each panoramic video segment, generate a two-dimensional map, segment the path of the panoramic video based on the nodes, and store the panoramic video sequence based on the path segments;

[0020] a position logic map module configured to locate a frame sequence of the panoramic video by using a mapping relationship between a two-dimensional map position and a number of panoramic video frames, and a mapping relationship between user physical space information and a position in a virtual space;

[0021] The panoramic video module is configured to continuously select frames for playback according to the frame sequence.

[0022] As an optional implementation, it also includes a data management module for storing and managing node information, path information, and physical space information of users when roaming.

[0023] As an optional implementation, the node information includes the node serial number, node name, and latitude and longitude information of the actual geographical location of the node;

[0024] The path information includes the path length information, the starting node number, the end point number and the corresponding video storage path;

[0025] The physical space information includes the position and orientation of the space where the user is located.

[0026] As an optional implementation, the two-dimensional map module includes:

[0027] The node position generation module is configured to map the node onto a two-dimensional plane based on the relationship between the node's latitude and longitude information and the screen resolution;

[0028] The path mapping module is configured to query the database for the start node and the end node of each path segment, generate a path between the two nodes, and connect the two nodes on a two-dimensional plane, which is represented as a two-dimensional path between the nodes;

[0029] The virtual scene position representation module is configured to receive the user's actual position information and locate the user's position in the virtual scene by calculating the frame number information of the corresponding path.

[0030] As an optional implementation, the location logic diagram module includes:

[0031] The node connection relationship storage module is configured to use an adjacency matrix to store connection information between nodes, establish an adjacency matrix by searching for the largest node number through a traversal of nodes, and find whether a path exists by using the node number. If so, the length of the path is stored in the adjacency matrix; if not, the length is set to the initial value;

[0032] The fast search module is configured to use a hash table to search two nodes to check whether there is a path between them.

[0033] As an optional implementation manner, the head-mounted display device interaction module includes:

[0034] The redirection module is configured to calculate the radius and curvature of the largest circle that can be accommodated based on the actual space area. During the roaming process, the translation gain, rotation gain, and curvature gain are dynamically introduced to guide the user's walking path to the circle.

[0035] The safety edge reset module is configured to rotate the viewing angle of the panoramic video 180° when the user walks to the edge of the safety range, using an improved 2:1 rotation algorithm, so that the user is back in the safety range and can continue to complete the roaming of the path.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention uses only panoramic video to enable users to walk around various scenes in a limited space while wearing a virtual reality device (such as HTC Vive, VR all-in-one device, etc.). This method is low-cost, simple, efficient, and easy to operate.

[0038] The present invention adopts a redirection algorithm that guides to a circle without a predetermined path and improves the 2:1 rotation algorithm at the boundary to enhance the immersive experience.

[0039] The present invention supports users to independently create roaming scenes and has greater flexibility.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 is a system flow chart of this embodiment;

[0043] FIG2( a ) shows a partial panoramic video naming method in an embodiment of the present invention;

[0044] FIG2( b ) shows the configuration of node location information in the SQLite database described in the embodiment of the present invention;

[0045] FIG2( c ) is a configuration of path data information corresponding to the panoramic video sequence of FIG2( a ) in the SQLite database according to an embodiment of the present invention;

[0046] Figure 3 It is a location logic map generated according to the SQLite database information in the implementation of the present invention;

[0047] Figure 4 A visual illustration of the mapping between the number of panoramic video frames and the position logic diagram described in the implementation of the present invention;

[0048] Figure 5 A two-dimensional plan view of a panoramic video sequence corresponding to the configuration of node information in the implementation of the present invention;

[0049] FIG6( a ) is a diagram showing a user about to collide with a physical boundary and the user's perspective in the Reset mechanism described in an embodiment of the present invention.

[0050] FIG6( b ) is a diagram showing the user turning around and the user's perspective in the Reset mechanism described in the embodiment of the present invention.

[0051] Figure 7 This is an explanation of predicting user roaming intention at a node based on a head-mounted display device as described in the implementation of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0055] The example of wearing an HTC VIVE head-mounted display to perform a real walking tour of a campus panoramic video is used for illustration. Of course, in other implementation examples, other head-mounted display devices can be worn to perform a real walking tour of other scenes.

[0056] A real walking and roaming system based on panoramic video. The system supports users to upload panoramic videos according to their own needs or the needs of tourists, and the videos are numbered according to nodes. Then, based on the panoramic video sequence, users can independently configure the location information of nodes (i.e., intersections) in the SQLite database and the path information of each panoramic video segment. Finally, the system uses a longitude and latitude conversion algorithm to generate a location logic diagram and a two-dimensional map based on the node information. Users wearing HTC VIVE can walk and roam in a real physical space of 3.5m*3.5m. The system can support configuration by scenic area staff or tourists, allowing them to walk and roam in tourist attractions or cities without leaving home during the epidemic prevention and control period, which can effectively reduce the pressure of epidemic prevention and control.

[0057] Specifically, the real walking roaming system based on panoramic video consists of five main functional modules, including: data organization and management module, location logic diagram module, two-dimensional map module, head-mounted display device interaction module, and panoramic video module.

[0058] The data organization and management module is used for data configuration and data reading and writing in the SQLite database, and is mainly responsible for maintaining node data and path data;

[0059] The location logic map module generates a weighted undirected graph from the node data, visualizes the logical relationship between the nodes, and calculates the user's geographical location in the logic map through the position and direction information of the head-mounted display device.

[0060] The two-dimensional map module realizes the global presentation of the scene and locates the user's location in real time. During the walking roaming process, the user can experience a continuous roaming experience or a jumpy fast roaming experience after specifying a location according to their needs.

[0061] The head-mounted display device interaction module is mainly used to output the position information of the head-mounted display device in the physical space. Based on this position information, the above-mentioned position logic map module calculates the corresponding number of frames in the panoramic video for presentation, and realizes the detection of physical boundary collision. When a physical boundary collision is about to occur, the reset mechanism is called.

[0062] The panoramic video module is used to locate the number of frames calculated by the position logic map module and implement video preloading based on the prediction of the user's turning intention.

[0063] Using the above-mentioned panoramic video-based real walking roaming system, the following steps are included:

[0064] (1) The user selects the creation mode to upload the panoramic video shot in segments and number them according to the specified format.

[0065] (2) Complete the basic configuration of the database: Use SQLite database and Navicat database visualization software to perform simple database configuration, enter the node location information No de (node ​​number, node name, longitude position, latitude position) and video path information Pathdata (starting node, ending node, video storage path, path length).

[0066] (3) Open the position logic map module: initialize the logic map, establish the mapping relationship between the ID and the node, and query the path through the two nodes; at the same time, receive the user's position and angle information in the above-mentioned head-mounted display device interaction module, and update the user's position in the virtual scene.

[0067] (4) Open the 2D map module: After entering the system, there will be a simplified scene map in the upper right corner of the screen in the main scene. Click on the map, the 2D map will be enlarged, switch to the map interface, and a more detailed map model will appear; the cursor will track the user's position in real time and be displayed on the large and small maps simultaneously; after entering the large map, you can see the location names of each scene and the related paths. Click on any point on the location or path and the system will automatically jump to the corresponding scene.

[0068] (5) Turn on the head-mounted display device interaction module: Real-time detection of the user's position and orientation information when walking in the physical space, through the redirection mechanism, reduce the possibility of the user colliding with the physical boundary, and start calling the reset mechanism when the user is 0.5m away from the physical boundary to ensure the safety of the user's actual walking and roaming.

[0069] (6) Turn on the panoramic video module: Display the spherical "curtain" that plays the panoramic video in the scene, and use the position information and orientation information of the head-mounted display device to predict the user's walking roaming path and preload it into the scene to achieve seamless roaming at the node.

[0070] In step 1, the user selects either creation mode or roaming mode. If roaming mode is selected, the user can roam through pre-created scenes in the database. If creation mode is selected, the user can upload panoramic video sequences captured with a panoramic camera and name them according to a unified numbering format. The specific format is "initial node number - ending node number.mp4," which allows other modules to locate the corresponding panoramic video sequence based on the two node numbers.

[0071] In step 2, if the user selects Roaming Mode, step 2 can be skipped. If the user selects Creation Mode and uploads the captured panoramic video, the shooting node and shooting path information needs to be updated. Specifically, the intersection node data number and latitude and longitude information include the information shown in the table below.

[0072] Field Name type illustrate id int Node number name varchar Node name (e.g. restaurant, playground) longitude decimal longitude latitude decimal latitude

[0073] The longitude and latitude information can be obtained through the map online website.

[0074] The path information for shooting each panoramic video includes the path length, starting point, end point, and corresponding video path information as shown in the table below.

[0075] Field Name type illustrate length float Path length (approximately based on video length) start int Starting Node end int end vUrl text Corresponding video storage path

[0076] In step 3, the specific method includes:

[0077] (3-1) Initializing the logical map: An adjacency matrix is ​​used to store node connectivity information. To reduce system overhead, a method is added to traverse the nodes to find the largest node ID and use this ID to build the adjacency matrix. Initialize the adjacency matrix so that all values ​​stored in the matrix are NO_PATH. The node ID is used to determine whether a path exists. If so, the path length is stored in the adjacency matrix; otherwise, NO_PATH is set.

[0078] (3-2) Use id to query nodes and use nodes to query paths, which corresponds to the LgGraphOutput interface. Use a hash table to find two nodes to check whether there is a path between them. The hash table is used to process and represent key-value pairs like (key, value). Keys are usually used for fast lookups. How to create a hash table:

[0079] id-node table: key-value pair (node ​​id, node) (3)

[0080] id, id-pathdata table: key-value pair ((id, id), pathdata) (4)

[0081] (3-3) The location and angle are mapped using the user's position and angle information in physical space. Specifically, if the user automatically predicts a path by turning the helmet at a node, the predicted roaming path is loaded and the user's position is updated in real time. If the user has not switched paths at a node but has chosen another path by roaming, the path of the user's clicked location is loaded and updated in real time to the video frame corresponding to the user's clicked location. The position mapping relationship is as follows:

[0082] location=preLocation+rate*(x-tx) / length (5)

[0083] Here, rate is a fixed quantity used to determine the ratio of the video playback speed to the human walking speed. It needs to be verified through experiments. When rate is between [7.5, 10], it best matches the actual walking speed of a person.

[0084] like Figure 4 As shown in formula (5), x represents the user position in the current frame, and tx represents the user position in the previous frame. The user position change is obtained by calculating the difference between the user position in the current frame and the position in the previous frame in the system. The ratio of the user position change to the total distance can be obtained by multiplying it with rate and dividing it by the length of the current path. Then, by adding it to the number of video frames traveled in the previous frame, the ratio value of the current user position to the entire path can be obtained (between 0 and 1). It is used in the system to represent the position of the user in the panoramic logical map.

[0085] In step 4, the specific implementation method includes:

[0086] (4-1) Node location generation. The locations in the two-dimensional map use the button components in the UI. First, create a button as the parent object in the scene. In the C# script, refer to the parent object to copy the button components of the corresponding number of nodes, modify their respective coordinates, and add listeners. When the button is pressed, the corresponding data is output to the location logic diagram module, and the scene can be switched.

[0087] Since the database stores the longitude and latitude of each location, and when displayed in Unity, horizontal and vertical coordinates are needed to determine the location, the longitude and latitude are converted into plane coordinates with a node as the coordinate origin according to the following formula.

[0088]

[0089]

[0090] Where e represents the eccentricity of the location, and the calculation formula is:

[0091]

[0092] θ represents the latitude of the location, a represents the length of the semi-major axis of the Earth's ellipse, and b represents the length of the semi-minor axis of the Earth's ellipse.

[0093] When performing coordinate conversion, the converted coordinate position may exceed the canvas, which may cause the custom button and path to deviate from the center position. To address this, a certain constraint relationship is established between the point coordinates and the canvas in the script, making the origin the minimum horizontal and vertical coordinates of the point and positioning it at 3 / 4 of the lower left corner of the canvas. At the same time, the scaling of other points is adjusted to ensure that the point is always within the central 3 / 4 area of ​​the canvas, thereby achieving the scalability of the two-dimensional map module.

[0094] (4-2) Path. The path uses the Slider component, and its implementation is similar to that of the node. The coordinates of the Slider use the node list in the node script. The midpoint of the starting point and the end point is taken as the coordinate data of the Slider. The direction of the path is calculated at the position passing through the starting point and the end point. The nodes are directly matched through rotation. The scaling of the Slider is related to the straight-line distance between the starting point and the end point. Since the Slider member variable value can directly represent the proportion of the user's click position in the path, if the Slider is selected as the path, the value can be directly output to the logical position map to adjust the user's position more finely. The path and nodes are dynamically generated through the database, thus ensuring the scalability of the system.

[0095] (4-3) In this embodiment, the user's position in the real-time virtual scene is represented by a cursor. The cursor uses the image component in the UI, and the cursor pattern supports user selection. The script connects to the position logic module, receives the user's position in real time, and updates the cursor coordinates and rotation angle. Specifically: obtain position information from the position logic module, locate the starting point a and end point b of the user's path using the data in PathData, further determine the user's coordinate range, obtain the proportion p of the user's path using location, and finally use the following formula to calculate the cursor coordinate f x and f y :

[0096] f x =a x +p*(b x -a x ) (9)

[0097] f y =a y +p*(b y -a y ) (10)

[0098] Angle represents the angle relative to the positive direction of the video. The function that converts Euler angles to quaternions can be directly applied to the cursor rotation.

[0099] In step 5, the specific implementation method includes: using LightHouse positioning technology to track the HTC VIVE's position in physical space and employing a circle-oriented redirection algorithm to enable roaming in a small space. Based on the actual space area of ​​3.5m*3.5m, a radius of 1.75m is calculated for the largest circle that can fit within the space. During the subsequent roaming process, translation gain, rotation gain, and curvature gain are dynamically introduced to guide the user, continuously directing their walking path to the circle. Furthermore, a reset mechanism ensures the safety of the user's roaming. As shown in Figure 6, an improved method based on a 2:1 rotation algorithm is used. When the user is about to reach the safety boundary, the system prompts the user to turn back and face the safety boundary. However, at this point, the user's perspective will be in the opposite direction of the original direction of travel. Therefore, the material ball representing the panoramic video is simultaneously rotated 180° to ensure that the user's roaming perspective remains unchanged and the roaming path continues within the safe range.

[0100] In step 6, the user's roaming path is predicted based on the user's perspective. The specific process includes: when the user reaches a node, the path will be automatically switched based on the user's perspective. The basis for switching is the angle between the user's perspective and each node. Figure 7 As shown, each node records the path information of its adjacent nodes, and obtains the angle of each path at the node turning point with the node as the coordinate origin and the forward direction as the zero-degree angle through actual measurement or open maps (such as Baidu Maps, Gaode Maps, etc.). The user's viewing direction is compared with the directions of each angle, and the path with the closest angle is selected as the user's next roaming path. However, since the user's viewing angle data comes from the head-mounted display device, the reference coordinate system of the angle is not consistent with the reference coordinate system for calculating the node angle. Therefore, the deflection angle from the head-mounted display device is converted into the angle between the starting point and the x-axis direction. In fact, the deflection angle (angle around the y-axis) in the head-mounted display device data represents the angle between the user's viewing angle and the negative direction of the panoramic video. The angle between the negative direction of the video and the x-axis direction with the starting point as the origin can also be obtained using AngleCaculator, denoted as α, and the angle θ finally used for judgment:

[0101] θ=f(rot y )-α (11)

[0102] where rot y Indicates the yaw angle of the head-mounted display device around the y-axis.

[0103] Since the deflection angle range recorded by HTC VIVE is [-180°, 180°], and the angle calculation range of AngleCaculator is [0°, 360°], the function f is needed to realize the conversion between the two bases.

[0104] by Figure 7 For example, the OB path initially has an angle of 30° with the user's original direction of travel, the OD path initially has an angle of 90° with the user's original direction of travel, the OA path initially has an angle of 180° with the user's original direction of travel, and the OC path initially has an angle of -90° with the user's original direction of travel. When the user is about to reach the intersection, the user's viewing direction θ = 75°. By comparison, the user's next roaming path is predicted to be OD.

[0105] As a typical embodiment, Figure 1 As shown, the system flow chart of this embodiment:

[0106] (1) Open.

[0107] (2) The user selects creation mode or roaming mode.

[0108] (3) Determine whether the mode selected by the user is the creation mode. If so, jump to step (4); otherwise, jump to step (8).

[0109] (4) The user uploads the panoramic video sequence and names it according to the specified format.

[0110] (5) The user configures the SQLite database to update the data.

[0111] (6) Automatically generate location logic diagram.

[0112] (7) Mapping of two-dimensional map geographic information and panoramic video frame numbers.

[0113] (8) Determine whether the roaming node has been reached. If yes, jump to step (9); otherwise, jump to step (10).

[0114] (9) Predict the user’s roaming intention and load the corresponding panoramic video.

[0115] (10) Determine whether the user's position in the physical space is about to touch the physical boundary. If so, jump to step (11); otherwise, jump to step (12).

[0116] (11) Call the Reset mechanism to ensure the user's safety and to continue roaming on the path after turning around.

[0117] (12) Determine whether roaming is to be terminated. If so, exit the system; otherwise, jump to step (8).

[0118] As shown in Figure 2(a), when users upload panoramic videos, they need to name them according to the specified naming method. This allows the system to accurately query and load the panoramic videos through nodes when reading them. Figure 2(b) shows the user's update and configuration of node information in the aforementioned SQLite database. The key information is the node ID number and the node's latitude and longitude information. Figure 2(c) shows the path information of each panoramic video, mainly including the start and end nodes of the video recording process and the video duration information.

[0119] like Figure 3 As shown, according to the node information and panoramic video path information in the database of FIG2 , a position logic diagram of this embodiment is generated, wherein the weight is the video length of each panoramic video segment.

[0120] like Figure 5 As shown, based on Figure 3 Generate a two-dimensional map including the panoramic video path, and use the actual two-dimensional map as the underlying layer to provide users with an overall preview effect. By clicking the path position in the two-dimensional map, users can quickly switch the path.

[0121] As shown in Figure 6(a), when the user is about to collide with the boundary of the physical space, the system will give a sound prompt and call the Reset mechanism, so that the user's perspective after turning around and entering the safe area (as shown in Figure 6(b) right) remains consistent with the previous perspective (as shown in Figure 6(a) right), completing the continued walking roaming of the path.

[0122] Figure 4 、 7 This has been explained in the above steps (3-3) and step 6 respectively and will not be repeated here.

[0123] The panoramic video-based real walking roaming system and method provided in the above embodiments can be implemented and have broad application prospects.

[0124] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0128] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A real walking roaming method based on panoramic video, characterized by: The following steps are involved: Obtain a panoramic video of the area to be roamed, where each panoramic video constitutes an independent path sequence; Recording the location information of each node and the start and end points and length of each panoramic video segment, generating a two-dimensional map, segmenting the path of the panoramic video based on the nodes, and storing the panoramic video sequence based on the path segments; Generate an undirected weighted graph based on the information of each node, save the connection information between the nodes through the adjacency matrix, initialize all the values ​​in the adjacency matrix to zero, and find out whether the path exists by the sequence number of the node. If it exists, update the corresponding element in the adjacency matrix to store the length of the path. If it does not exist, the corresponding element remains zero; By mapping the position on the two-dimensional map to the number of frames in the panoramic video, and the mapping between the user's physical space information and the position in the virtual space, the frame sequence of the panoramic video is located, and the frames are continuously selected for playback according to the frame sequence; An improved method based on a 2:1 rotation algorithm is used. When the user is about to reach the safety boundary, the user is prompted to turn around and face the safety boundary. However, at this time, the user's perspective will be in the opposite direction of the original travel direction, so the material ball presented by the panoramic video is rotated 180° at the same time to ensure that the user's roaming perspective remains unchanged and the roaming path continues within the safe range.

2. The method for real walking and roaming based on panoramic video according to claim 1, characterized in that: Methods for obtaining panoramic videos of the area to be roamed include users creating and shooting panoramic videos themselves or retrieving panoramic videos of existing scenes.

3. The method for real walking roaming based on panoramic video according to claim 1 is characterized in that: The mapping relationship between the map position and the panoramic video frame number is to determine the starting node and the ending node of each path, generate a path between the two nodes, and connect the two nodes on a two-dimensional plane, which is represented as a two-dimensional path between the nodes.

4. The method for real walking and roaming based on panoramic video according to claim 1, characterized in that: It also includes predicting the user's roaming path based on the user's perspective. The specific process includes: when the user arrives at a node, the path is switched according to the user's perspective. The basis for switching is the angle between the user's perspective and each node. Each node records the path information of its adjacent nodes. The angles of each path at the node turning point with the node as the coordinate origin and the forward direction as zero degrees are obtained. The user's perspective direction is compared with the directions of each angle, and the path with the closest angle is selected as the user's next roaming path.

5. A real walking roaming system based on panoramic video, characterized by: include: The head-mounted display device interaction module is configured to display the rendered image, determine the user's physical space information, and implement roaming using a redirection algorithm; The head-mounted display device interaction module includes: The redirection module is configured to calculate the radius and curvature of the largest circle that can be accommodated based on the actual space area. During the roaming process, the translation gain, rotation gain, and curvature gain are dynamically introduced to guide the user's walking path to the circle. The safety edge restart module is configured to rotate the panoramic video 180° when the user reaches the edge of the safety range, using an improved 2:1 rotation algorithm, so that the user is back in the safety range and can continue to complete the roaming path; A panoramic video acquisition module is configured to acquire a panoramic video of the area to be roamed, where each panoramic video constitutes an independent path sequence; a two-dimensional map module configured to record the location information of each node and the start and end points and length of each panoramic video segment, generate a two-dimensional map, segment the path of the panoramic video based on the nodes, and store the panoramic video sequence based on the path segments; a position logic map module configured to locate a frame sequence of the panoramic video by using a mapping relationship between a two-dimensional map position and a number of panoramic video frames, and a mapping relationship between user physical space information and a position in a virtual space; The position logic diagram module includes: The node connection relationship storage module is configured to use an adjacency matrix to store connection information between nodes, establish an adjacency matrix by searching for the largest node number through a traversal of nodes, and find whether a path exists by using the node number. If so, the length of the path is stored in the adjacency matrix; if not, the length is set to the initial value; The fast search module is configured to use a hash table to find two nodes to check whether there is a path between them; The panoramic video module is configured to continuously select frames for playback according to the frame sequence.

6. The panoramic video-based real walking and roaming system according to claim 5, characterized in that: It also includes a data management module for storing and managing node information, path information, and physical space information when the user roams; Alternatively, the node information may include a node serial number, a node name, and latitude and longitude information of the actual geographical location of the node; The path information includes the path length information, the starting node number, the end point number and the corresponding video storage path; The physical space information includes the position and orientation of the space where the user is located.

7. The panoramic video-based real walking and roaming system according to claim 5, characterized in that: The two-dimensional map module includes: The node position generation module is configured to map the node onto a two-dimensional plane based on the relationship between the node's latitude and longitude information and the screen resolution; The path mapping module is configured to query the database for the start node and the end node of each path segment, generate a path between the two nodes, and connect the two nodes on a two-dimensional plane, which is represented as a two-dimensional path between the nodes; The virtual scene position representation module is configured to receive the user's actual position information and locate the user's position in the virtual scene by calculating the frame number information of the corresponding path.

Images