A Cloud VR data transmission method and system based on edge computing storage

Abstract

The present invention relates to a Cloud VR data transmission method and system based on edge computing storage, which is used to transmit the image data of a panoramic image to a user-end device. The method includes: using a four-sided pyramid projection method to map and calculate the panoramic image, and dividing each panoramic image into multiple video versions with different viewing angles according to the viewpoint projection for storage; obtaining and dynamically predicting the three-dimensional motion data of the user-end device and calculating the user's viewing angle information; according to the user's viewing angle information, selecting the video version of the corresponding viewing angle, and performing a pyramid mapping operation on the video version to obtain stereoscopic image data, calculating the viewpoint position movement vector corresponding to the viewpoint switching, and transmitting the stereoscopic image data to the user-end device. Compared with the prior art, the present invention has the advantages of reducing the required bandwidth by more than 60%, lossless viewing quality within the field of view, avoiding image lag caused by network delay and network, and avoiding image loss caused by network jitter.

Description

Technical Field

[0001] The present invention relates to the field of Cloud VR data transmission technology, and in particular to a Cloud VR data transmission method and system based on edge computing storage. Background Art

[0002] The evolution of local VR to Cloud VR has become an inevitable trend. Cloud VR that supports cloud rendering has great value in improving user experience, reducing user consumption VR costs, protecting VR content copyrights, and popularizing VR commercial scenarios. After Cloud VR has clouded computing and rendering, due to the huge amount of data in VR, the transmission part will become the bottleneck of the entire system, so it is crucial to reduce the amount of data transmitted. There is a large redundancy in the complete transmission of panoramic images, but only transmitting images within the field of view will cause delays or black screens due to network delays and jitters, greatly affecting the viewing experience. Summary of the invention

[0003] The purpose of the present invention is to provide a Cloud VR data transmission method and system based on edge computing storage in order to overcome the defects of the above-mentioned prior art that the complete transmission of panoramic images has large redundancy, and may cause delays or black screens due to network delays and jitters, which greatly affects the viewing experience.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A Cloud VR data transmission method based on edge computing storage is used to transmit panoramic image data to a user terminal device, and the method comprises the following steps:

[0006] S1: The panorama is mapped and calculated using a tetrahedral projection method, and each panorama is divided into multiple video versions with different viewing angles according to the viewpoint projection for storage;

[0007] S2: Acquire and dynamically predict the three-dimensional motion data of the user terminal device and calculate the user's viewing angle information;

[0008] S3: According to the user perspective information, a video version of a corresponding viewing angle is selected, and a pyramid mapping operation is performed on the video version to obtain stereoscopic image data, and a viewpoint position movement vector corresponding to the perspective switching is calculated and the stereoscopic image data is transmitted to the user terminal device.

[0009] Furthermore, the step S2 specifically includes the following steps:

[0010] S21: Acquire and dynamically predict the view matrix of the user terminal device, and perform only rotation and inversion and transposition processing, wherein the view matrix includes a rotation matrix multiplied by a translation matrix, and is used to convert the world coordinate system of the object into the camera coordinate system;

[0011] S22: Calculate the three-dimensional direction vector of the viewing angle of the user terminal device in the current playback frame.

[0012] Furthermore, the step S3 specifically includes the following steps:

[0013] S31: acquiring a video version of a corresponding field of view angle according to the three-dimensional direction vector;

[0014] S32: generating a video picture group according to the video version and the current play pointer, and performing a pyramid mapping operation on the video picture group to obtain stereoscopic image data;

[0015] S33: Determine whether the viewpoint version switch occurs between the previous and next frames. If so, calculate the viewpoint position movement vector corresponding to the viewpoint switch, and move the viewpoint of the current frame in the stereoscopic image data in the corresponding direction. Otherwise, do not continue processing, and finally transmit the final stereoscopic image data to the user-end device.

[0016] Furthermore, if the viewpoint version of the previous and next frames is switched, the specific operation process is as follows:

[0017] Get the viewing direction of the previous frame and the viewpoint center of the current frame, and move the viewpoint center of the current frame to coincide with the viewing direction of the previous frame.

[0018] Furthermore, the mapping calculation of the panoramic image by using a tetrahedral projection method includes a mapping method based on the longitude and latitude of a globe, and the mapping method based on the longitude and latitude of a globe includes:

[0019] At the equator, starting from 0 degrees, a viewpoint is selected every 30 degrees, and there are 12 viewpoint versions on the equator; at 30 degrees south and 30 degrees north, starting from 0 degrees, a viewpoint is selected every 60 degrees, and there are 6 viewpoints at 30 degrees south and 30 degrees north; at 90 degrees south and 90 degrees north, starting from 0 degrees, a viewpoint is selected every 180 degrees, and there are 2 viewpoints at 90 degrees south and 90 degrees north; finally, there is a viewpoint at each of the North and South Poles, thus 12+6+6+2+2+1+1=30 different viewpoint versions.

[0020] Furthermore, step S1 also includes: numbering the video version and storing it; step S31 specifically includes: obtaining the number of the video version of the corresponding field of view angle according to the three-dimensional direction vector.

[0021] Furthermore, the calculation process of obtaining the serial number of the video version corresponding to the field of view angle is specifically as follows:

[0022] The boundaries are 15 degrees, 45 degrees, and 75 degrees north and south latitude, and the longitude line is the middle longitude line between two horizontal viewpoints. If the viewpoint moves within 15 degrees north and south and within 15 degrees left and right longitude, no viewpoint switching operation will occur. Otherwise, a viewpoint switching operation will occur, and the viewpoint switching operation is the number of the video version after the viewpoint moves.

[0023] Furthermore, in step S2, the dynamic prediction of the three-dimensional motion data of the user terminal device and the calculation of the user viewing angle information are specifically as follows:

[0024] The dead reckoning method is used to retain the viewpoint 3D data of the previous frame and perform linear prediction with the viewpoint 3D data of the current frame to obtain the viewpoint 3D data of the next frame, and the user viewing angle information is calculated based on the viewpoint 3D data of the next frame.

[0025] Furthermore, after the panorama is mapped and calculated by using a tetrahedral pyramid projection method, a square panorama is obtained, and the pyramid mapping operation includes:

[0026] Square Panorama Figure 4 The corner points are all set to P, and the midpoints of the four sides are set to A, B, C, and D respectively. P corresponds to the vertex of the regular tetrahedron, and A, B, C, and D correspond to the four corners of the square at the bottom of the regular tetrahedron, so as to perform pyramid mapping operations.

[0027] The present invention also provides a Cloud VR data transmission system based on edge computing storage, which is used to transmit panoramic image data to a user terminal device. The system includes an edge computing storage server, a client host and a user terminal device; the data processing process of the system includes the following steps:

[0028] S1: The edge computing storage service uses a tetrahedral pyramid projection method to map the panorama, and divides each panorama into multiple video versions with different field of view angles for storage according to the viewpoint projection;

[0029] S2: The client host obtains and dynamically predicts the three-dimensional motion data of the user-end device and calculates the user's perspective information, and finally transmits the user's perspective information to the edge computing storage server;

[0030] S3: The edge computing storage server selects a video version with a corresponding field of view according to the user perspective information, and sends it to the client host. The client host performs a pyramid mapping operation on the video version to obtain stereoscopic image data, calculates the viewpoint position movement vector corresponding to the perspective switching, and transmits the stereoscopic image data to the user-end device.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] In order to ensure that users can still watch high-quality video images when the viewing angle position changes, the present invention processes panoramic images according to viewing angles, and processes panoramic images into multiple versions according to the central viewpoint, and selects the corresponding version for transmission based on the viewing angle information fed back by the user; on the content preparation side, a full-view uneven quality code stream is prepared for each viewing angle, and high quality is used in the viewing area; low-quality coding is used in the non-viewing area. According to the user's current viewing angle posture position, the corresponding viewing angle file is requested; when the viewing angle changes when the head rotates, the viewing angle file corresponding to the new viewing angle is requested; the technical solution of the present invention has the following advantages as a whole:

[0033] (1) The required bandwidth can be reduced by more than 60%;

[0034] (2) The viewing quality is lossless within the field of view;

[0035] (3) Avoid image lag and network delay caused by network delay;

[0036] (4) Avoid image loss caused by network jitter. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a three-dimensional right-handed coordinate system diagram of the present embodiment;

[0038] Figure 2 3D coordinate transformation diagram of this embodiment;

[0039] In the figure, vertex data is vertex data, object coordinates is object coordinates, clipcoordinates is clip coordinates, divide by w is divide by w, normalized device coordinates is standardized device coordinates, and window coordinates is window coordinates;

[0040] Figure 3 It is a flow chart of the present invention;

[0041] Figure 4 This is a schematic diagram of dynamic prediction based on viewing angle in this embodiment;

[0042] Figure 5 This is the tetrahedron / pyramid mapping diagram of this embodiment;

[0043] Figure 6 This is the viewpoint division diagram of 30 in this embodiment;

[0044] Figure 7 This is a diagram of the edge computing storage framework of this embodiment;

[0045] In the figure, central cloud refers to the central cloud, edge of cloud refers to the edge cloud, and user refers to the user;

[0046] Figure 8 This is the Fov transmission scheme diagram of this embodiment;

[0047] In the figure, server is the server, viewpoint is the viewpoint, network is the network, and fov render client is the fov rendering client;

[0048] Fig. 9 It is a schematic diagram of a tetrahedron / pyramid mapping of this embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0052] Example 1

[0053] This embodiment provides a Cloud VR data transmission method based on edge computing storage, which is used to transmit panoramic image data to a user terminal device. The panoramic image is processed into multiple versions according to the central viewpoint, and the corresponding version is selected for transmission based on the viewing angle information fed back by the user; on the content preparation side, a full-view uneven quality code stream is prepared for each viewing angle, and high quality is used in the viewing area; low-quality encoding is used in the non-viewing area. According to the user's current viewing posture position, the corresponding viewing angle file is requested. When the viewing angle changes due to the head rotation, the viewing angle file corresponding to the new viewing angle is requested.

[0054] The method specifically comprises the following steps:

[0055] S1: The panorama is mapped and calculated using a tetrahedral projection method, and each panorama is divided into multiple video versions with different viewing angles according to the viewpoint projection for storage;

[0056] The mapping calculation of the panorama using the tetrahedral projection method includes a mapping method based on the longitude and latitude of the globe, and the mapping method based on the longitude and latitude of the globe includes:

[0057] At the equator, starting from 0 degrees, a viewpoint is selected every 30 degrees, and there are 12 viewpoint versions on the equator; at 30 degrees south and 30 degrees north, starting from 0 degrees, a viewpoint is selected every 60 degrees, and there are 6 viewpoints at 30 degrees south and 30 degrees north; at 90 degrees south and 90 degrees north, starting from 0 degrees, a viewpoint is selected every 180 degrees, and there are 2 viewpoints at 90 degrees south and 90 degrees north; finally, there is a viewpoint at each of the North and South Poles, thus 12+6+6+2+2+1+1=30 different viewpoint versions.

[0058] S2: Acquire and dynamically predict the three-dimensional motion data of the user terminal device and calculate the user's viewing angle information;

[0059] Step S2 specifically includes the following sub-steps:

[0060] S21: Acquire and dynamically predict the view matrix of the user terminal device, and perform rotation and inversion and transposition processing, where the view matrix includes a rotation matrix multiplied by a translation matrix, and is used to convert the world coordinate system of the object into the camera coordinate system;

[0061] The dynamic prediction process is as follows:

[0062] The dead reckoning method is used to retain the viewpoint 3D data of the previous frame and perform linear prediction with the viewpoint 3D data of the current frame to obtain the viewpoint 3D data of the next frame, and the user viewing angle information is calculated based on the viewpoint 3D data of the next frame.

[0063] S22: Calculate the three-dimensional direction vector of the viewing angle of the user terminal device in the current playback frame.

[0064] S3: According to the user's viewing angle information, a video version of the corresponding viewing angle is selected, and a pyramid mapping operation is performed on the video version to obtain stereoscopic image data, and the corresponding viewpoint position movement vector of the viewing angle switching is calculated and the stereoscopic image data is transmitted to the user terminal device.

[0065] Step S3 specifically includes the following sub-steps:

[0066] S31: Obtain a video version of a corresponding field of view angle according to the three-dimensional direction vector;

[0067] S32: generating a video picture group according to the video version and the current play pointer, and performing a pyramid mapping operation on the video picture group to obtain stereoscopic image data;

[0068] S33: Determine whether the viewpoint version switch occurs between the previous and next frames. If so, calculate the viewpoint position movement vector corresponding to the viewpoint switch, and move the viewpoint of the current frame in the stereoscopic image data in the corresponding direction. Otherwise, do not continue processing, and finally transmit the final stereoscopic image data to the user-end device.

[0069] If the viewpoint version of the previous and next frames is switched, the specific operation process is as follows:

[0070] Get the viewing direction of the previous frame and the viewpoint center of the current frame, and move the viewpoint center of the current frame to coincide with the viewing direction of the previous frame.

[0071] Step S1 also includes: numbering the video versions and storing them; step S31 specifically includes: obtaining the number of the video version corresponding to the field of view angle according to the three-dimensional direction vector.

[0072] The specific calculation process for obtaining the serial number of the video version corresponding to the field of view angle is as follows:

[0073] The boundaries are 15 degrees, 45 degrees, and 75 degrees north and south latitude, and the longitude line is the middle longitude line between two horizontal viewpoints. If the viewpoint moves within 15 degrees north and south and within 15 degrees left and right longitude, no viewpoint switching operation will occur. Otherwise, a viewpoint switching operation will occur, and the viewpoint switching operation is the number of the video version after the viewpoint moves.

[0074] After the panorama is mapped and calculated using a tetrahedral pyramid projection method, a square panorama is obtained. The pyramid mapping operation includes:

[0075] Square Panorama Figure 4 The corner points are all set to P, and the midpoints of the four sides are set to A, B, C, and D respectively. P corresponds to the vertex of the regular tetrahedron, and A, B, C, and D correspond to the four corners of the square at the bottom of the regular tetrahedron, so as to perform pyramid mapping operations.

[0076] This embodiment also provides a Cloud VR data transmission system based on edge computing storage, which is used to transmit image data of a panoramic image to a user terminal device. The system includes an edge computing storage server, a client host, and a user terminal device;

[0077] The system processes panoramic images according to viewing angles. To ensure that users can still watch high-quality video images when their viewing angles change, the server needs to make timely adjustments based on the user's location information. The server first processes the panoramic image into 30 versions based on the central viewpoint, and selects the corresponding version for transmission based on the viewing angle information fed back by the user. On the content preparation side, a full-view uneven quality code stream is prepared for each viewing angle, and high quality is used in the viewing area. Low-quality encoding is used in non-viewing areas. The terminal requests the corresponding viewing angle file from the server based on the user's current viewing angle posture position. When the viewing angle changes due to the head rotation, the terminal requests the viewing angle file corresponding to the new viewing angle from the server.

[0078] The data processing process of the system includes the following steps:

[0079] S1: The edge computing storage service uses a tetrahedral pyramid projection method to map the panorama, and divides each panorama into multiple video versions with different field of view angles for storage according to the viewpoint projection;

[0080] The specific steps in this embodiment are: the panoramic image is mapped and calculated by using a tetrahedral projection method, and each panoramic image is divided into 30 different FOV versions for storage according to the viewpoint projection, that is, each panoramic video has 30 different FOV video versions in the edge storage server;

[0081] It includes the following sub-steps:

[0082] S11, performing tetrahedral projection on the ERP panorama to generate a square panorama;

[0083] S12, taking the longitude and latitude of the globe as an example to explain the division: at the equator, starting from 0 degrees, a viewpoint is selected every 30 degrees, and there are 12 viewpoint versions on the equator; at the 30 degrees of north and south dimensions, starting from 0 degrees, a viewpoint is selected every 60 degrees, and there are 6 viewpoints on the 30 degrees of north and south dimensions respectively; at the 90 degrees of north and south dimensions, starting from 0 degrees, a viewpoint is selected every 180 degrees, and there are 2 viewpoints on the 90 degrees of north and south dimensions respectively; finally, there is one viewpoint at each of the North and South Poles, thus 12+6+6+2+2+1+1=30 different viewpoint versions are composed.

[0084] 30 The version numbers are sorted as follows:

[0085] From the North Pole to the South Pole, starting from 0 degrees longitude and moving in the right-hand direction, they are numbered 0, 1, 2...29.

[0086] S2: The client host obtains and dynamically predicts the three-dimensional motion data of the user-end device and calculates the user's perspective information, and finally transmits the user's perspective information to the edge computing storage server;

[0087] The specific steps in this embodiment are: the client host obtains and dynamically predicts the three-dimensional motion data of the user's VR helmet in real time and calculates the user's perspective information, and finally transmits the user's perspective information to the edge computing storage server;

[0088] It includes the following sub-steps:

[0089] S21, obtain the view matrix of the VR helmet, and set "rotation only" and inverse and transposition processing;

[0090] S22, the spatial coordinate system is established according to the right-hand system, and the three-dimensional direction vector of the viewing angle of the current playback frame is calculated;

[0091] S23, the Fov direction vector is transmitted to the edge computing storage server.

[0092] The schematic diagram of dynamic prediction based on viewing angle is as follows Figure 4 As shown, the prediction mainly includes eight main directions: up, down, left, right, upper right, lower right, upper left, and lower left. The dynamic prediction method is as follows:

[0093] Perform dynamic viewpoint prediction based on machine learning, using dead reckoning. Keep the viewpoint 3D data of the previous frame, and linearly predict the viewpoint 3D data of the next frame with the viewpoint 3D data of the current frame.

[0094] S3: The edge computing storage server selects the video version with the corresponding field of view according to the user's viewing angle information and sends it to the client host. The client host performs pyramid mapping operation on the video version to obtain stereoscopic image data, calculates the corresponding viewpoint position movement vector for the viewing angle switching, and transmits the stereoscopic image data to the user-end device.

[0095] The specific steps in this embodiment are: the edge computing server selects the corresponding version video GOP according to the user perspective information and sends it back to the client host, and the client host calculates the corresponding viewpoint position movement vector of the perspective switch and transmits the image to the VR helmet.

[0096] It includes the following sub-steps:

[0097] S31, calculating the corresponding version number according to the Fov direction vector;

[0098] The process of calculating the version number is as follows:

[0099] The latitudes of 15, 45, and 75 degrees are used as boundaries, and the longitude line is the middle longitude line between two horizontal viewpoints. If the viewpoint moves within 15 degrees north and south and within 15 degrees left and right of the longitude, the viewpoint switching operation will not occur. If the viewpoint moves from 0 degrees longitude and 0 dimensions to 16 degrees longitude and 0 dimensions, it should be switched to the version with 30 degrees longitude and 0 dimensions as the viewpoint center.

[0100] S32, sending the next video GOP to the client host according to the serial number and the current play pointer, and sending it in a format where each GOP has 10 frames, and the client host then performs a pyramid mapping operation on the received video GOP in a square panorama format;

[0101] The pyramid mapping process occurs as follows Fig. 9 As shown, the specific mapping method is as follows:

[0102] The four corner points of the square are set to P, and the midpoints of the four sides are set to A, B, C, and D. P corresponds to the vertex of the regular tetrahedron, and A, B, C, and D correspond to the four corners of the square at the bottom of the regular tetrahedron.

[0103] S33, determine whether the viewpoint version switch occurs between the previous and next frames. If yes, calculate the viewpoint position movement vector corresponding to the viewpoint switch, move the viewpoint of the current frame in the corresponding direction, and finally transmit the image to the VR helmet.

[0104] The edge storage framework of the present invention is a one-to-one server-client format, but is not limited to this. The number of clients can be increased to form a one-to-many format. Multiple edge computing storage devices can be set up on this basis to communicate with the clients and be managed by a central server.

[0105] The specific implementation process of the Cloud VR data transmission system based on edge computing storage in this embodiment is described below:

[0106] 1) The edge computing storage server uses a tetrahedral pyramid projection method to map and calculate the panorama. Each panorama is divided into 30 different FOV versions for storage according to the viewpoint projection. That is, each panoramic video has 30 different FOV video versions in the edge storage server;

[0107] 11) Perform tetrahedral projection on the ERP panorama to generate a square panorama. The projection effect is as follows: Figure 5 As shown, the rectangular panorama is mapped into a square panorama;

[0108] 111) For any pixel point G(m,n) of the plane image, determine the current surface;

[0109] 112) Determine the corresponding mapping coordinates according to which face the pixel belongs to, and convert the two-dimensional coordinates G(m,n) into three-dimensional coordinates P(x,y,z) in the cube space;

[0110] 113) Obtain the spherical coordinates M of the point in the spherical coordinate system according to the mapping formula;

[0111] 114) Reversely derive the position N of the point in the spherical coordinate system in the image according to the formula for generating the spherical panorama;

[0112] 115) Determine whether the coordinate of N is an integer;

[0113] 116) If yes, directly obtain the pixel values ​​of the N points and assign them to the pixel points of the cube plane image;

[0114] 117) Otherwise, bidirectional linear interpolation is used to obtain the pixel values ​​of the N points and assign them to the pixel points of the cube plane image.

[0115] 12) Divide the square panorama into viewpoint versions, such as Figure 6 As shown;

[0116] 121) Take the latitude and longitude of the globe as an example to explain the division: starting from 0 degrees on the equator, a viewpoint is selected every 30 degrees, and there are 12 viewpoint versions on the equator;

[0117] 122) On the 30 degrees north-south dimension, starting from 0 degrees, select a viewpoint every 60 degrees. There are 6 viewpoints on the 30 degrees north-south dimension.

[0118] 123) On the 90-degree north-south dimension, starting from 0 degree, select a viewpoint every 180 degrees. There are 2 viewpoints on each of the 90-degree north-south dimension.

[0119] 124) Finally, the north and south poles are each a viewpoint. Thus, 12+6+6+2+2+1+1=30 different viewpoint versions are constructed.

[0120] 13) Divide the viewpoint versions by number, such as Figure 6 As shown;

[0121] 131) From the North Pole to the South Pole, starting from 0 degrees longitude and moving in the right-hand direction, they are numbered 0, 1, 2, ... 29.

[0122] 14) The edge storage framework of this embodiment is a one-to-one server-client form, but it is not limited to this. The number of clients can be increased to form a one-to-many form. On this basis, multiple edge computing storages can be set up to communicate with the clients and be managed by a central server, such as Figure 7 shown.

[0123] 2) The client host obtains the 3D motion data of the user's VR helmet in real time and calculates the user's viewing angle information, and finally transmits the user's viewing angle information to the edge computing storage server;

[0124] 21) Get the view matrix of the VR helmet (m_mat4HMDPose), and perform corresponding rotate-only and related inverse and transposition processing;

[0125] 211) The function of the view matrix is ​​to convert the world coordinate system of the object into the camera coordinate system;

[0126] 212) View matrix = rotation matrix * translation matrix;

[0127] 213) Figure 1 As shown, the computer internal rendering establishes a right-handed coordinate system in space, that is, the right side is the x-axis, the back is the z-axis, and the top is the y-axis. The view matrix under the right-handed system is as follows:

[0128]

[0129] 214) The view matrix in the left-handed system is as follows:

[0130]

[0131] 215)eye(eyeX,eyeY,eyeZ) is the viewpoint, that is, the position of the camera in the world coordinate system;

[0132] 216)at(atX,atY,atZ) is the observation point, i.e. the observed target point, pointing in the direction of the camera;

[0133] 217) up (upX, upY, upZ) is the top direction, that is, determines which direction the camera is directly above;

[0134] 218) Zaxis is the viewpoint direction vector, that is, the direction vector at looking towards the eye;

[0135] 219)Yaxis and Xaxis are the direction vectors of the y and x axes of the world coordinate system;

[0136] 220) Setting the view matrix to rotate-only means that the camera can only perform rotation operations without translation, scaling, etc.; from the above, we can see that in the right hand system, we only need to set the ground 4 columns of the view matrix to (0, 0, 0, 1);

[0137] 221) The view matrix needs to be transposed in the right-handed system, but not in the left-handed system.

[0138] 22) The spatial coordinate system is established according to the right-hand system, and the three-dimensional direction vector of the current playback frame viewing angle is calculated;

[0139] 222) From 213) and 218), it can be seen that the Zaxis vector is the three-dimensional direction vector of the viewing angle to be obtained;

[0140] 223) The view matrix is ​​first negated and then multiplied by (0, 0, 1), and the first three components of the third row of the view matrix under the right hand system are taken out to form a new three-dimensional Fov direction vector;

[0141] 23) The Fov direction vector is transmitted to the edge computing storage server;

[0142] 24) While acquiring the view matrix of the current viewpoint in real time, dynamic viewpoint prediction based on machine learning is performed using dead reckoning.

[0143] 3) The edge computing server selects the corresponding version of the video GOP based on the user's perspective information and sends it back to the client host. The client host calculates the corresponding viewpoint position movement vector for the perspective switch and transmits the image to the VR helmet.

[0144] 31) Calculate the corresponding version number according to the Fov direction vector;

[0145] 311) The latitudes of 15, 45, and 75 degrees north and south are the boundaries;

[0146] 312) The longitude line is bounded by the middle longitude line between the two horizontal viewpoints;

[0147] 313) If the viewpoint moves within 15 degrees north-south dimension and within 15 degrees left and right longitude, the viewpoint switching operation will not occur. If the viewpoint moves from 0 degrees longitude and 0 degrees to 16 degrees longitude and 0 degrees, it should be switched to the version with 30 degrees longitude and 0 degrees as the viewpoint center.

[0148] 32) Send the next video GOP to the client host according to the number and the current play pointer, such as Figure 8 As shown;

[0149] 321) Divide each video stream into one-second GOPs and implement view-dependent adaptive bitrate streaming for playback. Every second (or less), determine the next stream to be acquired based on the user's network conditions, viewport direction, and the time of the next GOP;

[0150] 322) Run multiple iterations of the test in a controlled network environment, recording the angular distance from the viewport center to the viewer's direction, the time spent in each viewport, viewport resolution and switching latency, buffering time, and network bandwidth.

[0151] 33) Determine whether the viewpoint version switch occurs between the previous and next frames. If yes, calculate the viewpoint position movement vector corresponding to the viewpoint switch, move the viewpoint of the current frame in the corresponding direction, and finally transmit the image to the VR helmet;

[0152] 331) Figure 2 As shown, the 3D data is transformed by the model matrix, projection matrix and view matrix before being rendered onto the VR helmet.

[0153] 332)ModelViewMatrix:model matrix;

[0154] 333)ProjectionMatrix: projection matrix;

[0155] 334)ViewportTransform:view matrix;

[0156] 335)mModel=ModelViewMatrix*ProjectionMatrix*ViewportTransform: model projection view matrix;

[0157] 336) first obtain the viewing direction of the previous frame and the viewpoint center of the current frame, and move the viewpoint center of the current frame to coincide with the viewing direction of the previous frame;

[0158] 3361) Update viewpoint:

[0159] frameViewpoint.phi=(90-frameViewPointTable[viewPointIndex][0])*M_PI / 180;

[0160] frameViewpoint.theta=frameViewPointTable[viewPointIndex][1]*M_PI / 180;

[0161] 3362) The viewpoint center of the current frame moves to coincide with the viewpoint direction of the previous frame;

[0162] float sin_phi=sin(frameViewpoint.phi);

[0163] float cos_phi=cos(frameViewpoint.phi);

[0164] float sin_theta=sin(frameViewpoint.theta);

[0165] float cos_theta=cos(frameViewpoint.theta);

[0166] #define M(row,col)mModel[col*4+row]

[0167] M(0,0) = cos_theta;

[0168] M(0,1)=sin_theta*sin_phi;

[0169] M(0,2)=sin_theta*cos_phi;

[0170] M(0,3)=0.0;

[0171] M(1,0)=0.0;

[0172] M(1,1)=cos_phi;

[0173] M(1,2)=-sin_phi;

[0174] M(1,3)=0.0;

[0175] M(2,0) = -sin_theta;

[0176] M(2,1)=cos_theta*sin_phi;

[0177] M(2,2) = cos_theta*cos_phi;

[0178] M(2,3)=0.0;

[0179] M(3,0)=0.0;

[0180] M(3,1)=0.0;

[0181] M(3,2)=0.0;

[0182] M(3,3)=1.0;

[0183] #undef M

[0184] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A Cloud VR data transmission method based on edge computing storage, used to transmit panoramic image data to a user terminal device, characterized in that: The method comprises the following steps: S1: The panorama is mapped and calculated using a tetrahedral projection method, and each panorama is divided into multiple video versions with different viewing angles according to the viewpoint projection for storage; S2: Acquire and dynamically predict the three-dimensional motion data of the user terminal device and calculate the user's viewing angle information; S3: According to the user perspective information, a video version of a corresponding viewing angle is selected, and a pyramid mapping operation is performed on the video version to obtain stereoscopic image data, and a viewpoint position movement vector corresponding to the perspective switching is calculated and the stereoscopic image data is transmitted to the user terminal device.

2. According to the Cloud VR data transmission method based on edge computing storage according to claim 1, it is characterized in that: The step S2 specifically includes the following steps: S21: Acquire and dynamically predict the view matrix of the user terminal device, and perform only rotation and inversion and transposition processing, wherein the view matrix includes a rotation matrix multiplied by a translation matrix, and is used to convert the world coordinate system of the object into the camera coordinate system; S22: Calculate the three-dimensional direction vector of the viewing angle of the user terminal device in the current playback frame.

3. According to claim 2, a Cloud VR data transmission method based on edge computing storage is characterized in that: The step S3 specifically comprises the following steps: S31: acquiring a video version of a corresponding viewing angle according to the three-dimensional direction vector; S32: generating a video picture group according to the video version and the current play pointer, and performing a pyramid mapping operation on the video picture group to obtain stereoscopic image data; S33: Determine whether the viewpoint version switch occurs between the previous and next frames. If so, calculate the viewpoint position movement vector corresponding to the viewpoint switch, and move the viewpoint of the current frame in the stereoscopic image data in the corresponding direction. Otherwise, do not continue processing, and finally transmit the final stereoscopic image data to the user-end device.

4. According to claim 3, a Cloud VR data transmission method based on edge computing storage is characterized in that: If the viewpoint version of the previous and next frames is switched, the specific operation process is as follows: Get the viewing direction of the previous frame and the viewpoint center of the current frame, and move the viewpoint center of the current frame to coincide with the viewing direction of the previous frame.

5. According to the Cloud VR data transmission method based on edge computing storage as described in claim 3, it is characterized in that: The method of mapping the panoramic image using a tetrahedral projection method includes a mapping method based on the longitude and latitude of a globe, and the mapping method based on the longitude and latitude of a globe includes: At the equator, starting from 0 degrees, a viewpoint is selected every 30 degrees, and there are 12 viewpoint versions on the equator; at 30 degrees south and 30 degrees north, starting from 0 degrees, a viewpoint is selected every 60 degrees, and there are 6 viewpoints at 30 degrees south and 30 degrees north; at 90 degrees south and 90 degrees north, starting from 0 degrees, a viewpoint is selected every 180 degrees, and there are 2 viewpoints at 90 degrees south and 90 degrees north; finally, there is a viewpoint at each of the North and South Poles, thus 12+6+6+2+2+1+1=30 different viewpoint versions.

6. A Cloud VR data transmission method based on edge computing storage according to claim 5, characterized in that: Step S1 also includes: numbering the video version and storing it; step S31 specifically includes: obtaining the number of the video version corresponding to the field of view angle according to the three-dimensional direction vector.

7. A Cloud VR data transmission method based on edge computing storage according to claim 6, characterized in that: The calculation process of obtaining the serial number of the video version corresponding to the field of view angle is specifically as follows: The boundaries are 15 degrees, 45 degrees, and 75 degrees north and south latitude, and the longitude line is the middle longitude line between two horizontal viewpoints. If the viewpoint moves within 15 degrees north and south and within 15 degrees left and right longitude, no viewpoint switching operation will occur. Otherwise, a viewpoint switching operation will occur, and the viewpoint switching operation is the number of the video version after the viewpoint moves.

8. The Cloud VR data transmission method based on edge computing storage according to claim 1 is characterized in that: In step S2, the dynamic prediction of the three-dimensional motion data of the user terminal device and the calculation of the user viewing angle information are specifically as follows: The dead reckoning method is used to retain the viewpoint 3D data of the previous frame and perform linear prediction with the viewpoint 3D data of the current frame to obtain the viewpoint 3D data of the next frame, and the user viewing angle information is calculated based on the viewpoint 3D data of the next frame.

9. The Cloud VR data transmission method based on edge computing storage according to claim 1, characterized in that: After the panorama is mapped and calculated by using a tetrahedral pyramid projection method, a square panorama is obtained. The pyramid mapping operation includes: Set the four corner points of the square panorama to P, and the midpoints of the four sides to A, B, C, and D respectively. P corresponds to the vertex of a regular tetrahedron, and A, B, C, and D correspond to the four corners of the square at the bottom of the regular tetrahedron, so as to perform pyramid mapping operation.

10. A Cloud VR data transmission system based on edge computing storage, used to transmit panoramic image data to a user terminal device, characterized in that: The system includes an edge computing storage server, a client host and a user terminal device; the data processing process of the system includes the following steps: S1: The edge computing storage service uses a tetrahedral pyramid projection method to map the panorama, and divides each panorama into multiple video versions with different field of view angles for storage according to the viewpoint projection; S2: The client host obtains and dynamically predicts the three-dimensional motion data of the user-end device and calculates the user's perspective information, and finally transmits the user's perspective information to the edge computing storage server; S3: The edge computing storage server selects a video version with a corresponding field of view according to the user perspective information, and sends it to the client host. The client host performs a pyramid mapping operation on the video version to obtain stereoscopic image data, calculates the viewpoint position movement vector corresponding to the perspective switching, and transmits the stereoscopic image data to the user-end device.

Images