A panoramic scene roaming interaction method and system based on motion vehicle driving

Abstract

The application provides a panoramic scene roaming interaction method and system based on motion vehicle driving, relates to the fields of virtual reality technology and human-computer interaction technology, and the user wears a virtual reality device, rides a motion vehicle along a preset physical path, and performs panoramic scene roaming interaction on pre-prepared virtual content during driving, comprising: acquiring real-time motion state information of the motion vehicle; based on the motion state information, combining user cognitive style and physical turning radius, real motion of the motion vehicle is redirected and mapped into a virtual video path, corresponding panoramic video frames and panoramic video orientation information are obtained; the panoramic video frames and the panoramic video orientation information are sent to the virtual reality device for real-time analysis and rendering, and roaming interaction of the user on the virtual content is realized. The application enables the user to ride a vehicle to roam a wider virtual space in a limited physical space, and realizes high-immersion vehicle-mounted VR experience.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology and human-computer interaction technology, and in particular to a panoramic scene roaming interaction method and system based on motion vehicle drive. Background Art

[0002] With the emergence of panoramic cameras, panoramic virtual scene (panoramic scene for short) roaming, as a popular form of virtual reality application, provides a new roaming experience, allowing users to enjoy the scenery of all parts of the world without leaving home. Therefore, it has been widely used in cultural tourism, education and science popularization, spiritual healing and other fields.

[0003] At present, users mainly experience pre-shot panoramic scenes by passively watching panoramic videos, or roaming in panoramic scenes through preset interactive hotspots. This type of interaction has limited freedom and it is difficult to achieve active roaming of users in panoramic virtual scenes. Some works use the interaction method of natural walking to drive the synchronous playback of panoramic videos, making users feel that they have achieved an experience similar to their daily walk in the park. However, this method is not suitable for people with limited mobility and also has certain limitations such as safety. Using sports vehicles such as cars and wheelchairs instead of natural walking (i.e., in-vehicle VR) can solve this problem.

[0004] In-car VR experiences not only assist users with exercise but also effectively enhance immersion through the sense of motion generated during movement, providing users with a more realistic virtual experience and, to a certain extent, alleviating the discomfort caused by the mismatch between vision and proprioception (motion sickness). For this reason, a growing number of studies are using vehicles as a medium for VR roaming interactions, with widespread applications in entertainment and virtual roaming. However, practical applications of this technology still face numerous challenges, such as the large physical space requirements—the need for a proportional mapping between real and virtual space—which has limited the widespread adoption and promotion of in-car VR technology.

[0005] Therefore, existing panoramic scene roaming technology is limited by key issues such as non-isometric mapping of virtual and real space and insufficient dynamic interaction, and cannot achieve a highly immersive in-vehicle VR experience in a limited physical space. Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes a panoramic scene roaming interaction method and system based on motion vehicle drive, which enables users to roam in a wider virtual space by riding a motion vehicle in a limited physical space, and achieve a highly immersive in-vehicle VR experience.

[0007] According to some embodiments, the present invention adopts the following technical solutions:

[0008] A panoramic scene roaming interaction method based on a motion vehicle drive, a user wearing a virtual reality device, riding a motion vehicle along a preset physical path, and roaming interaction with a pre-made virtual content in a panoramic scene during the ride, comprising:

[0009] Real-time motion state information of the motion vehicle is acquired, including position information and orientation data;

[0010] Based on the motion state information, the real motion of the motion vehicle is redirected and mapped into a virtual video path in combination with the user's cognitive style and the physical turning radius, to obtain corresponding panoramic video frames and panoramic video orientation information;

[0011] The panoramic video frames and panoramic video orientation information are sent to the virtual reality device for real-time analysis and rendering, to realize the user's roaming interaction with the virtual content, wherein during the roaming interaction, the panoramic video playback speed is adjusted in real time based on a dynamic translation gain.

[0012] According to some embodiments, the present application adopts the following technical solutions:

[0013] A panoramic scene roaming interaction system based on a motion vehicle drive, a user wearing a virtual reality device, riding a motion vehicle along a preset physical path, and roaming interaction with a pre-made virtual content in a panoramic scene during the ride, comprising:

[0014] A real-time acquisition module configured to acquire real-time motion state information of the motion vehicle, including position information and orientation data;

[0015] A redirection module configured to redirect and map the real motion of the motion vehicle into a virtual video path based on the motion state information in combination with the user's cognitive style and the physical turning radius, to obtain corresponding panoramic video frames and panoramic video orientation information;

[0016] A roaming interaction module configured to send the panoramic video frames and panoramic video orientation information to the virtual reality device for real-time analysis and rendering, to realize the user's roaming interaction with the virtual content, wherein during the roaming interaction, the panoramic video playback speed is adjusted in real time based on a dynamic translation gain.

[0017] According to some embodiments, the present application adopts the following technical solutions:

[0018] A computer program product comprising a computer program, which, when executed by a processor, implements the panoramic scene roaming interaction method based on a motion vehicle drive.

[0019] According to some embodiments, the present application adopts the following technical solutions:

[0020] A non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the motion vehicle driving-based panoramic scene roaming interaction method.

[0021] According to some embodiments, the present application adopts the technical scheme as follows:

[0022] An electronic device, comprising a processor, a memory and a computer program, wherein the processor is connected with the memory, and the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the motion vehicle driving-based panoramic scene roaming interaction method.

[0023] Compared with the prior art, the present application has the following beneficial effects:

[0024] The present application provides a motion vehicle driving-based panoramic scene roaming interaction method and system, which uses a motion vehicle to perform non-equal-distance mapping of a physical space position to a virtual space based on real-time position information and orientation data and other motion state information, realizes immersive virtual roaming in a limited physical space, and solves the problems of insufficient non-equal-distance mapping and dynamic interaction between virtual and real spaces, and has the following advantages:

[0025] Firstly, the present application can accurately calculate the motion state of a user in a real space by integrating an IMU and a multi-source data acquisition module of a microcontroller, combining a personalized curvature gain mapping algorithm, and mapping it to a virtual space in real time, realizing a more natural and coherent space redirection effect, realizing roaming experience in a small space, and significantly improving the immersion of the user.

[0026] Secondly, the present application supports users to independently shoot panoramic video content, constructs a virtual roaming path based on a directed graph structure, dynamically plans a navigation route combining a shortest path algorithm, gives the user higher path freedom and controllability, and at the same time, introduces a dynamic gain adjustment mechanism, which can automatically adjust the roaming speed according to the user's preference for the virtual scene, further optimizing the visual coherence and interaction fluency.

[0027] Thirdly, in view of the problem that the traditional redirection technology is prone to failure in space structures such as "back" shaped corridors, the present application introduces a path segmentation and arc transition strategy to effectively avoid collision risks.

[0028] Finally, the present application supports a two-person collaborative roaming mode, and adopts a modular design, has good expansibility and platform compatibility, and can be widely applied to various immersive virtual reality scenes such as education and training, cultural tourism, health care and nursing, and psychological intervention in the future. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings, which constitute a part of this specification, are incorporated herein to provide further understanding of the present application, and are incorporated for illustration of the exemplary embodiments of the present application and to explain the present application without imposing a limitation on the present application.

[0030] Figure 1 Actual application effect diagram for example 1;

[0031] Figure 2 Method flow chart for example 1;

[0032] Figure 3 Carrier control design and communication structure diagram for example 1;

[0033] Figure 4 Panoramic video shooting requirement diagram for example 1;

[0034] Figure 5 Panoramic video road map structure for example 1;

[0035] Figure 6 Virtual path selection method diagram for example 1;

[0036] Figure 7 Motion carrier driven redirection mapping algorithm flow chart for example 1;

[0037] Figure 8 Mapping relationship diagram for example 1;

[0038] Figure 9 Corridor corner environment collision-free constraint diagram for example 1;

[0039] Figure 10 "U" shaped corridor path segment diagram for example 1;

[0040] Figure 11 "U" shaped corridor mapping algorithm diagram for example 1. DETAILED DESCRIPTION

[0041] The present application will be further described below with reference to the drawings and embodiments.

[0042] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0043] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0044] Embodiment 1

[0045] In an embodiment of the present application, a panoramic scene roaming interaction method based on a motion vehicle drive is provided. A user wears a virtual reality device and rides a motion vehicle along a predetermined physical path. During the ride, the user performs panoramic scene roaming interaction on pre-made virtual content, including:

[0046] Step S1: Real-time motion state information of the motion vehicle is obtained, including position information and orientation data.

[0047] Step S2: Based on the motion state information, the user's cognitive style, and the physical turning radius, the real motion of the motion vehicle is redirected and mapped into a virtual video path to obtain corresponding panoramic video frames and panoramic video orientation information.

[0048] Step S3: The panoramic video frames and panoramic video orientation information are sent to the virtual reality device for real-time analysis and rendering, realizing the user's roaming interaction with the virtual content. During the roaming interaction, the panoramic video playback speed is adjusted in real time based on a dynamic translation gain.

[0049] As an embodiment, the panoramic scene roaming interaction method based on a motion vehicle drive allows the motion trajectory of a user riding a vehicle in a real environment to be mapped into a virtual reality environment, and allows the virtual video path to be inconsistent with the physical path. Real motion refers to the actual movement of a vehicle ridden by a user in a physical space (such as autonomous driving, remote remote control driving by others, or manual driving). Virtual motion refers to the motion trajectory perceived by a user wearing a virtual reality device (such as a VR headset) in a virtual scene. Through this method, the user travels along a curved path, but visually experiences a smooth journey along a straight path, thereby improving immersion and spatial consistency.

[0050] Specifically, the method installs an inertial measurement unit (IMU, such as MPU-6050) and a microcontroller unit (such as ESP32, model not limited) on the moving vehicle to obtain the current motion state information of the vehicle and feedback control in real time. After obtaining the actual motion state of the vehicle, the redirection mapping algorithm is used to calculate the panoramic video frame number to be played and the corresponding orientation, and a virtual picture consistent with the user's motion state is rendered, so that the continuous motion of the vehicle presents a continuous and natural virtual roaming experience in vision, realizing synchronous roaming in virtual and real spaces.

[0051] It is worth mentioning that the virtual content used in the embodiment supports users to take pictures by themselves through the panoramic camera and makes personalized path selection based on the panoramic video road network structure, which has good scalability and adaptability. In addition, in order to further enhance the interactive experience of users, the embodiment also introduces a virtual AI guide function and designs a panoramic video playback rate adjustment mechanism based on dynamic translation gain to improve the sense of immersion and participation.

[0052] The hardware and modules involved in realizing the above functions are briefly described as follows:

[0053] (1) Moving vehicle: used for carrying users and performing actual moving operations in virtual roaming. The moving vehicle is a two-seater electric platform with a maximum load of not less than 100 kg, supporting stepless speed regulation in the range of 0-0.3 m / s, and having multi-directional control capability, including forward, backward, left turn and right turn direction control.

[0054] (2) Sensor acquisition module: used for real-time acquisition of motion state information of the vehicle. The module includes an inertial measurement unit (IMU, such as MPU-6050) and a microcontroller unit (such as ESP32, model not limited), which can obtain position information and orientation data of the vehicle and other motion state information in real time.

[0055] (3) Communication module: used for realizing data communication between the moving vehicle and the VR end. The module is based on Wi-Fi network and transmits motion state data through Socket communication protocol to ensure real-time and synchronization of system interaction.

[0056] (4) Control module: used for controlling the basic moving behavior of the moving vehicle. The module outputs high and low level signals through the GPIO pin of the ESP32 microcontroller to realize the control of the forward, backward, left turn and right turn actions of the vehicle.

[0057] Based on the above content, the specific implementation steps are explained and described in combination with the drawings.

[0058] The embodiment provides a panoramic scene redirection roaming method based on a motion carrier drive, and basic equipment of the method comprises a virtual reality device (such as PICO 4) worn by a user and a carrier (an electric carrier capable of carrying two people, with a maximum load of 100 kg and a maximum running speed of 0.3 m / s) used for physical motion.

[0059] Figure 1 It is shown that a user rides a carrier to travel along a preset physical path (a red dashed line), and the vehicle can travel automatically by remote control or by setting a fixed steering wheel angle. At the same time, the virtual reality device synchronously renders panoramic video content, and a frame picture is updated in real time along with movement of the vehicle, so that the user always sees a coherent virtual scene in the virtual reality environment, as shown in Figure 2 It is shown that the method comprises the following steps:

[0060] Step S101: carrier control design

[0061] By integrating an IMU and an ESP32 on the carrier, real-time acquisition and feedback control of a motion state (including position information and orientation data) are realized. Data transmission is performed by using a local area network or a server, and the content to be transmitted is position information of a virtual space, and orientation information and video frame numbers (only for panoramic video) and the like are attached, so that information synchronization is finally realized.

[0062] Specifically, as shown in Figure 3 An IMU sensor is mounted on an ESP32 module and welded to the carrier, so as to collect motion data of the IMU in real time; by using a built-in Wi-Fi module, the ESP32 transmits data to a vehicle-mounted VR end program by using Socket communication; this process ensures that the VR end can acquire the motion state of the vehicle in real time and accurately dynamically adjust the motion state, and the VR end can transmit control information to the ESP32 module by using Socket communication, so as to control the height of a pin, thereby realizing control of simple movement of the carrier in front, back, left and right directions.

[0063] Step S102: virtual content production

[0064] A user records content by using a panoramic camera, and defines a virtual video path based on a video road network structure, so as to realize an individualized immersive experience, and high-quality immersive visual content is provided for subsequent virtual roaming; of course, the virtual content of the embodiment has a form of expression that includes but is not limited to panoramic video, and also includes a three-dimensional model scene.

[0065] Specifically, the step comprises two sub-steps:

[0066] Step S1021: acquisition of panoramic video

[0067] As shown in Figure 4As shown in the figure, a panoramic video sequence is required to shoot a complete virtual video path using a panoramic camera (for example, Insta360 X4). The total duration T is 300s, the frame rate r is 25fps, the speed v is 0.5m / s, t is the time, f is the number of frames, s is the distance, S is the total distance, and F is the total number of frames. The following requirements must be met during the shooting process:

[0068] (1) Viewpoint height and stability requirements: The panoramic camera is mounted on a tripod and fixed on a vehicle, approximately 1.2 meters above the ground, simulating the viewpoint height of a person sitting in it.

[0069] (2) Shooting path requirements: In order to ensure the comprehensiveness and aesthetics of the virtual tour, a relatively spacious and beautiful path should be selected, and sharp turns or sections with large curvature should be avoided to avoid obvious visual deviations that may cause motion sickness.

[0070] (2) Camera viewing angle requirements: When shooting, the camera must be aligned with the road surface; although the route may bend or turn, the camera's forward direction should always remain parallel to the shooting route.

[0071] Step S1022: Panoramic video road map structure construction and path selection

[0072] like Figure 5 As shown, based on the panoramic video sequence obtained in step S1021 and combined with the road network information of the real physical scene, a directed graph model is used to construct a panoramic video road map structure to support users to freely choose paths in a virtual environment.

[0073] The panoramic video road map structure is represented by a directed graph G = (N, E), where N represents the set of all nodes, each node represents an intersection or key point of interest; E represents the set of edges between nodes, each edge represents a path from one node to another, and stores corresponding information such as path length and video address.

[0074] It is worth noting that in this structure, every two adjacent nodes contain bidirectional edges, that is, at nodes Ni and Ni+1, there is both path information from Ni→Ni+1 and path information from Ni+1→Ni, ensuring that users can roam in any direction.

[0075] After obtaining the panoramic video road map structure, there are multiple virtual video path selection strategies, such as path selection based on the shortest path, path selection based on user interest point weights, and navigation based on the least repeated path.

[0076] like Figure 6As shown in Stage 1 in the figure, this embodiment selects the shortest path strategy as an example for detailed description to minimize the user's viewing time and burden. This embodiment uses the traditional Dijkstra algorithm to calculate the optimal path based on the starting point and end point set by the user, that is, to obtain the virtual video path to be roamed.

[0077] The algorithm starts from the starting point and gradually expands the search range using a priority queue, always giving priority to the node with the lowest current cumulative path cost until it reaches the end point. Specifically, when the algorithm is initialized, the path cost of the starting point is set to 0, and the path cost of all other nodes is set to infinity. Subsequently, in each iteration, the node with the lowest current cumulative path cost is selected for expansion, and the path costs and predecessor node information of its adjacent nodes are updated. When the end point node is popped out of the priority queue, the algorithm terminates immediately and generates the shortest path sequence by backtracking the predecessor nodes.

[0078] like Figure 6 As shown in stage 2 in , after the virtual video path is obtained, the virtual video sequence corresponding to the virtual video path is obtained according to the video address stored in the panoramic video path map structure.

[0079] In addition, this embodiment designs an interactive UI so that the user can customize the start and end points of the virtual video path on the interactive interface.

[0080] Step S103: Redirection mapping algorithm based on IMU motion compensation

[0081] Based on curvature gain technology, personalized curvature parameters are set in combination with the user's cognitive style (such as field-dependent or field-independent), mapping real movement to the corresponding panoramic video frame and orientation in the virtual video path, and compensating for rotation errors in real time through IMU data, so that users perceive movement as a straight line in the virtual space, ensuring visual continuity and directional consistency, and realizing non-isometric mapping between virtual space and real space.

[0082] The algorithm flow is as follows Figure 7 The specific sub-steps are as follows:

[0083] Step S1031: setting a suitable real physical path based on the curvature gain perception thresholds of individuals with different cognitive styles and the turning performance of the vehicle. In this embodiment, a circular physical path is used as an example.

[0084] Step S1032: Figure 8 As shown, the initialization parameters: center , radius R, total number of frames F, user's previous position , the cumulative distance traveled , the total length of the actual physical path S, the current direction , current location ;

[0085] Step S1033: Calculate the total length of the physical path S (assuming that the vehicle drives N circles in the real environment, and C is the circumference of the circular path) :

[0086]

[0087] Step S1034: Obtain the current position P of the vehicle through the IMU current And the direction of the vehicle , call this algorithm in a loop to get the panoramic video frame number f and the direction corresponding to each position for rendering, specifically:

[0088] (1) According to the current position , the center position and the previous time position , the angle walked through at the previous and next two times is obtained :

[0089]

[0090]

[0091] (2) Calculate the arc length walked through :

[0092]

[0093] Where, is the cumulative distance traveled, is the arc length walked through at the previous and next two times, and R is the radius.

[0094] (3) Map to the video frame of the panoramic video sequence, which is expressed by the formula:

[0095]

[0096] Where F is the total number of video frames, k is the mapping rate, is the arc length walked through, and S is the total length of the real physical path.

[0097] (4) Use the current direction of the vehicle to correct the direction of the panoramic video β, specifically:

[0098] The initial direction of the vehicle is 0, the current direction of the vehicle is , and the rotation deviation of the current vehicle is . The direction of the panoramic video deviates due to the change in the direction of the vehicle, and based on the current rotation deviation of the vehicle , the direction of the panoramic video is adjusted to to realize the deviation compensation of the panoramic video.

[0099] The embodiment is not only applicable to the mapping of the circular path, but also to the "back" shaped corridor for the actual roaming area. Since the mapping relationship of the circular path and the back shaped path is different, the mapping relationship needs to be further explained.

[0100] Step S104: Redirection path mapping of the "back" shaped corridor

[0101] The embodiment first analyzes the feasibility of the vehicle driving in the corridor, and then designs a redirection path mapping algorithm for the "back" shaped corridor for the actual roaming area based on the above mapping, to ensure the accurate correspondence between the virtual panoramic video and the actual physical motion trajectory, which is specifically:

[0102] Step S1041: Determine whether the vehicle can smoothly and safely drive in the corridor

[0103] In order to ensure the smooth and safe driving of the vehicle in the corridor, it is necessary to avoid collision between the vehicle and the wall of the corridor during driving, that is, to convert the problem of whether the vehicle can smoothly and safely drive into the problem of whether there is a collision-free path, such as Figure 9 As shown in the figure, when turning in the corridor environment, three key reference points Q1, Q2 and Q3 need to be considered. During the turning maneuver, if the constraint conditions specified in the following formulas are met, the vehicle can smoothly pass through:

[0104]

[0105] Among them, the variables and respectively represent the horizontal distance and the vertical distance from the rotation center to the inner boundary of the obstacle, and r represents the distance from the rotation center to the center of the vehicle, represents the width of the horizontal channel, represents the width of the vertical channel, W represents the width of the vehicle, and L represents the length of the vehicle.

[0106] Further, under the premise that the ranges of and are known, according to the above formulas, the minimum and maximum turning radii , and the instantaneous rotation center position under these environmental constraint conditions can be calculated, wherein and are determined according to the environmental representation.

[0107] Finally, the ranges of , , are obtained, which are expressed by the formula as:

[0108]

[0109] That is to say, if there is a drivable area under the above three range constraints, which contains a feasible collision-free path, then the vehicle can drive smoothly and safely along the collision-free path.

[0110] In this embodiment, since the vehicle width W = 0.8m, the vehicle length L = 1m, the corridor width ,By calculating all possible combinations of rotation center points and ,the corresponding turning radius, a feasible collision-free path can be ,determined, that is, the vehicle can travel normally in the corridor.

[0111] Step S1042: Path segmentation processing

[0112] As Figure 10 As an example of the physical scene, the central axis of the physical scene (here expressed as the center line of the circular path, that is, the path with equal distance from the walls on both sides, such as Figure 10 The green dashed line (shown as the physical path) is used as the path. The overall path is primarily straight, but at turns, the vehicle's turning radius is limited by walls and spatial constraints, so an arc is used for smooth transitions. At the same time, to prevent the vehicle from getting too close to the wall, the corridor boundary is reduced inward by a distance d (d = 0.1m) when calculating the path. Overall, the core idea is to segment the path. The optimized motion trajectory (green dashed line) consists of four alternating straight line sub-paths (P1 / P3 / P5 / P7) and four arc sub-paths with a radius of R (P2 / P4 / P6 / P8). This allows the vehicle to complete roaming smoothly while ensuring driving safety and immersion.

[0113] Step S1043: Determine current location Which index path (i.e., the i-th sub-path) is in, and the distance from the starting point is calculated based on the position, that is, the distance from the current position to the circle , which can be expressed as:

[0114]

[0115] in, is the length of the j-th subpath, is the starting point of the i-th subpath, Used to calculate the distance between two locations.

[0116] Step S1044: Calculate the total distance traveled based on the cumulative number of laps q:

[0117]

[0118] Where l is the distance traveled, is the distance of the current position from the distance in this circle, q is the number of circles accumulated, is the length of each circle of the actual path.

[0119] Step S1045: mapping the video frames of the panoramic video sequence (F is the total number of video frames, k is the mapping rate):

[0120]

[0121] Step S1046: as shown in Figure 11 , the panoramic video is corrected towards using the vehicle orientation. .

[0122] Step S1047: when the vehicle reaches the end point or the panoramic video is played, the roaming ends.

[0123] Step S105: application and interaction

[0124] Introduce virtual tour guide service (elf image) in virtual roaming, integrate speech recognition (STT), natural language processing (such as based on large language model) and speech synthesis (TTS) technology, realize real-time voice commentary based on scene content and natural dialogue between user and virtual role, improve scene understanding and interaction, the specific steps are:

[0125] (1) Use STT technology to convert user's voice input into text in real time, so that it has editable and processable ability, which is convenient for subsequent analysis and understanding.

[0126] (2) Input the converted text into the large model for intelligent analysis. The model can understand user's intention, analyze scene context, and generate responses that conform to logic and grammar.

[0127] (3) Finally, use TTS technology to convert the text generated by the large model into speech, which is output by the three-dimensional virtual image.

[0128] According to the scene content, the gain intensity of the roaming rate is calculated, and the panoramic video playing speed is dynamically adjusted, so that the user can obtain a rhythm suitable and personalized navigation experience in the panoramic video playing process. Based on the dynamic translation gain, the playing speed control supports users to freely select the gain rate according to the demand, uses the interpolation method to smooth adjust the mapping frame number, realizes the dynamic adjustment of the translation gain, which is:

[0129] (1) Calculate the gain rate at the current time t:

[0130]

[0131] wherein, G(t) is the gain rate at the current time t, G(t) is the gain rate at the current time t, G(t) is the target gain rate, T is the interpolation time, t is the time during the interpolation process, and belongs to "0, T".

[0132] (2) Video frame number f Mapping:

[0133] Optionally, a multi-user synchronization module is supported, and the position information and the perspective state of the first user are shared through a network to realize immersive collaborative roaming experience between two or more users.

[0134] The above steps are a general method of panoramic scene roaming interaction based on motion vehicle driving. The method can realize vehicle driving-based redirected roaming, map a simple circular path or a zigzag path to a virtual video path, and support mapping of three-dimensional scenes in addition to panoramic videos. The method can be used for, but is not limited to, psychological intervention treatment, patient and elderly companion, infant education, and entertainment and relaxation.

[0135] To sum up, in order to realize vehicle driving-based immersive virtual roaming in a limited physical space, the embodiment uses an IMU and an ESP32 microcontroller to modify an interactive vehicle, realizes real-time acquisition and transmission of vehicle motion data. At the same time, based on interactive panoramic video road map technology, a virtual video path selection method is provided, so that users can shoot and select panoramic content materials, and then obtain personalized panoramic virtual video paths. In addition, in combination with the curvature gain perception threshold of users with different cognitive styles and the turning radius constraint of the vehicle, a vehicle-mounted VR curvature gain mapping algorithm based on IMU compensation is proposed, non-equal mapping of physical space positions to virtual space is realized, and thus immersive vehicle-mounted VR roaming experience is realized in a limited physical space. Finally, a complete vehicle-mounted VR roaming scheme of "vehicle control design-virtual content production-redirected roaming-application and interaction" is formed.

[0136] Embodiment 2

[0137] In an embodiment of the present application, a panoramic scene roaming interaction system based on motion vehicle driving is provided. A user wears a virtual reality device and rides a motion vehicle along a preset physical path. During the ride, panoramic scene roaming interaction is performed on pre-produced virtual content, including:

[0138] The real-time acquisition module is configured to acquire real-time motion state information of the motion vehicle, including position information and orientation data;

[0139] The redirection module is configured to: based on the motion state information, in combination with the user cognitive style and the physical turning radius, map the real motion of the motion carrier into a virtual video path to obtain corresponding panoramic video frames and panoramic video orientation information.

[0140] The roaming interaction module is configured to: send the panoramic video frames and the panoramic video orientation information to a virtual reality device for real-time analysis and rendering, so as to realize the roaming interaction of the user on the virtual content, wherein, in the roaming interaction process, the real-time adjustment is performed on the panoramic video playing speed based on the dynamic translation gain.

[0141] Embodiment 3

[0142] In an embodiment of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the panoramic scene roaming interaction method based on the motion carrier driving.

[0143] Embodiment 4

[0144] In an embodiment of the present application, a non-transitory computer readable storage medium is provided, which is used to store computer instructions, and the computer instructions, when executed by a processor, implement the panoramic scene roaming interaction method based on the motion carrier driving.

[0145] Embodiment 5

[0146] In an embodiment of the present application, an electronic device is provided, comprising a processor, a memory and a computer program, wherein the processor is connected with the memory, and the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the panoramic scene roaming interaction method based on the motion carrier driving.

[0147] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for implementing the functions specified in one flow or multiple flows and / or blocks Figure 1 The device for implementing the functions specified in one flow or multiple flows and / or blocks

[0148] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0149] The above describes the specific embodiments of the present application in conjunction with the drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A panoramic scene roaming interaction method based on motion vehicle driving, characterized in that: Users wear virtual reality devices and ride on a motion vehicle along a preset physical path. During the ride, they interact with pre-made virtual content in a panoramic scene, including: Obtain real-time motion status information of the moving vehicle, including position information and orientation data; Based on motion state information, combined with user cognitive style and physical turning radius, the real motion of the moving vehicle is redirected and mapped into the virtual video path, obtaining the corresponding panoramic video frame and panoramic video orientation information; The panoramic video frames and panoramic video orientation information are sent to the virtual reality device for real-time analysis and rendering, enabling users to roam and interact with the virtual content. During the roaming interaction process, the panoramic video playback speed is adjusted in real time based on the dynamic translation gain. The real motion redirection of the moving vehicle is mapped to the virtual video path to obtain the corresponding panoramic video frame and panoramic video orientation information, specifically: A circular physical path is set based on the curvature gain perception threshold of individuals with different cognitive styles and the vehicle's turning performance; Calculate the current curvature radius and the center of the redirection gain circle based on the current position information and the set curvature gain parameters; Based on the curvature radius and the center of the redirection gain circle, the user's position in the real space is mapped to the corresponding coordinates in the virtual space, thereby achieving path redirection from the real space to the virtual space and obtaining the mapped virtual coordinates; Match the mapped virtual coordinates with the virtual video path nodes to determine the corresponding panoramic video frames, achieving synchronous playback of audio-visual content and user movement; Using the real-time motion state information provided by the inertial measurement unit, we compensate for the orientation error and correct the user's orientation perception in the virtual scene to ensure that it is consistent with the visual information. It also includes redirection mapping of non-circular physical paths, which optimizes the motion trajectory through path segmentation and arc transition strategies.

2. The method for interactive panoramic scene roaming based on motion vehicle driving according to claim 1, characterized in that: The virtual content is produced in the following steps: Obtain the panoramic video required for the complete virtual video path; Based on the acquired panoramic video and combined with the road network information of the real physical scene, a panoramic video road map structure is constructed and a virtual video path is generated.

3. The method for interactive panoramic scene roaming based on motion vehicle driving according to claim 1, characterized in that: The motion vehicle is provided with an inertial measurement unit and a microcontroller, which are respectively used for collecting motion state information of the motion vehicle and performing motion control of the motion vehicle.

4. The method for interactive panoramic scene roaming based on motion vehicle driving according to claim 1, characterized in that: The real-time adjustment of the panoramic video playback speed based on the dynamic translation gain is to smoothly adjust the mapping frame number through an interpolation method, dynamically adjust the translation gain, and remap the panoramic video frame based on the adjusted translation gain.

5. A panoramic scene roaming interactive system based on motion vehicle driving, characterized by: A panoramic scene roaming interaction method based on motion vehicle driving according to any one of claims 1 to 4 is adopted, wherein a user wears a virtual reality device and rides a motion vehicle along a preset physical path, and performs panoramic scene roaming interaction with pre-made virtual content during the driving process, including: The real-time acquisition module is configured to: acquire real-time motion state information of the motion vehicle, including position information and orientation data; The redirection module is configured to: based on the motion state information, combined with the user's cognitive style and physical turning radius, redirect the real motion of the moving vehicle into the virtual video path to obtain the corresponding panoramic video frame and panoramic video orientation information; The roaming interaction module is configured to: send panoramic video frames and panoramic video orientation information to the virtual reality device for real-time analysis and rendering, thereby enabling the user to roam around the virtual content. During the roaming interaction process, the panoramic video playback speed is adjusted in real time based on the dynamic translation gain.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for roaming and interacting in a panoramic scene based on motion vehicle driving as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the panoramic scene roaming interaction method based on motion vehicle driving as described in any one of claims 1 to 4 is implemented.

8. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement a panoramic scene roaming interaction method based on motion vehicle driving as described in any one of claims 1-4.

Images