Method for dynamically changing image deformation according to user position in virtual reality real shot video

Abstract

The invention discloses a method for dynamically changing image deformation according to a user position in a virtual reality real shot video, which adopts a Unity engine as an implementation platform, and comprises the following steps of: 1) projecting an image in the real shot video into a sky box of a sphere structure; and 2) modifying image rendering information in a Unity engine Shader shader according to the position of the user so as to dynamically change image deformation according to the position of the user. The image deformation can be dynamically changed according to the user position, the implementation mode is simple and convenient, and the user experience can be improved.

Description

Technical Field

[0001] The present invention relates to a method for dynamically changing image distortion according to the user's position in a virtual reality live-action video, which can be applied to fields such as VR videos and VR movies. Background Art

[0002] The application of virtual reality (VR) technology in the field of real-shot videos is mainly reflected in enhancing the shooting experience, providing new narrative methods, and improving the post-production process, etc. 360-degree video shooting: Using a dedicated 360-degree camera or stitching multiple cameras shot at fixed angles into a panoramic video; Real-time preview: During the shooting process, through VR technology, the captured images or virtual scenes are shown to the director or photographer in real time from the first-person perspective; Post-production integration: Video editing is carried out in a three-dimensional space, including but not limited to editing, adding special effects, color correction, etc.

[0003] VR technology has its unique advantages in the display of image content, but there are still certain limitations in its actual application:

[0004] Although the images captured by a 360-degree camera can bring a more free viewing angle to the audience, this only expands the selectable range of the audience's field of view. The equidistant images will break the immersive experience because they cannot synchronize with the changes in the user's position, and even cause negative effects such as motion sickness. In the scenario of real-time preview, the fixed camera position cannot reflect the deviation of the picture composition caused by the change of the viewing angle in the real environment, and it is impossible to accurately locate the influence area of adding special effects in the three-dimensional space during the post-production process. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for dynamically changing image distortion by considering the position and viewing angle information of the audience when playing a real-shot video in a virtual reality environment.

[0006] The purpose of the present invention is achieved by the following technical solutions: A method for dynamically changing image distortion according to the user's position in a virtual reality live-action video, which uses the Unity engine as the implementation platform, and includes the following steps:

[0007] Step 1) Project the images in the real-shot video onto the skybox of the spherical structure;

[0008] Step 2) Modify the image rendering information in the Unity engine Shader shader according to the user's position to achieve dynamically changing the image distortion according to the user's position.

[0009] The specific implementation method of Step 2) is as follows:

[0010] a) Obtain the user's initial position and continuously obtain their dynamic position in real-time. Compare the two positions to obtain the direction User_direction1 and distance distance1 of the user's movement. Then map the direction vector User_direction1 to the texture space, obtain its position coordinates (x, y) in the texture space, and use them as the scaling center point coordinates in the direction of the user's approach.

[0011] b) In the Shader shader, use two vector variables _MinifyCenter1 and _MinifyCenter2 to carry the scaling center point coordinates in the directions of the user's approach and departure respectively. Set the scaling factor variable _MinifyAmount1 as distance1 / R, the deformation radius _Radius as _MinifyAmount1 * 0.25, and the attenuation radius _Falloff as _MinifyAmount1, where R represents the set operation area radius, which is used to limit the user's movement range.

[0012] c) Establish a comparison vector i.dir, which represents the direction vector from the user's initial position to a certain pixel position. Similarly, map this direction vector to the texture space, and its position is represented as uv.

[0013] d) Adjust the position of the uv coordinates relative to the scaling center according to the scaling factors _MinifyAmount1 and _MinifyAmount2, thereby producing a scaling effect.

[0014] Magnification effect:

[0015] Assign the horizontal and vertical coordinates of _MinifyCenter1 to the variable center, let the difference vector delta = uv - center, and calculate the length dist of delta.

[0016] When dist < _Radius, perform the following operations:

[0017] Use the smoothstep function to return a smooth transition value factor between 0 and 1 according to the value of dist in the range of _Radius and _Radius - _Falloff.

[0018] Use the lerp function to perform linear interpolation between 1.0 and 1 + _MinifyAmount1 according to the value of factor, and then scale the difference vector delta according to the scaling factor determined by the lerp function to obtain the new uv coordinates relative to center.

[0019] Shrinking effect:

[0020] Let _MinifyAmount2 = -_MinifyAmount1;

[0021] Assign the horizontal and vertical coordinates of _MinifyCenter2 to the variable center, let the difference vector delta = uv - center, and calculate the length dist of delta;

[0022] When dist < _Radius, perform the following operations:

[0023] Through the smoothstep function, based on the value of dist in the range between _Radius and _Radius - _Falloff, return a smooth transition value factor between 0 and 1;

[0024] Use the lerp function to perform linear interpolation between 1.0 and 1 + _MinifyAmount2 according to the value of factor, and then scale the difference vector delta according to this scaling factor determined by the lerp function to obtain the new uv coordinates relative to center.

[0025] When the user moves in a certain direction, the present invention determines information such as the magnification center point and the minification center point by calculating the moving direction and mapping it to the texture space, and accesses the Shader shader of the Unity engine. Then, in the Shader shader, it realizes the dynamic change of image deformation according to the user's position.

[0026] In the Shader shader, the present invention sets MinifyAmount1 as distance1 / R so as to adjust the scaling degree according to the ratio of the user's approaching distance, sets the deformation radius, i.e., the influence range _Radius of the deformation effect, as MinifyAmount1 * 0.25 to confine the deformations in the approaching and departing directions within their respective half - sphere ranges, sets the falloff radius _Falloff directly equal to _MinifyAmount1 so that its influence area can cover the entire sphere, and then combines with the smoothstep function to slow down the overall scaling intensity, thereby achieving the effect of texture scaling and natural transition within a local area (limited by _Radius).

[0027] In step b), the abscissa in _MinifyCenter1 is x, and the ordinate is y. The abscissa and ordinate in _MinifyCenter2 are set to mathf.abs(1 - x) and y respectively according to the abscissa and ordinate in _MinifyCenter1. The mathf.abs function is used to obtain the absolute value of 1 - x. In this way, it is more convenient to directly set the vector information related to the scaling center point in the Shader shader for the away direction.

[0028] The position information used to represent the height of the VR headset in the dynamic position is generated by the user when the VR device accesses the Unity engine, or is generated by accessing the positioning device in the virtual reality device.

[0029] The radius of the operation area is determined by the user setting when the VR device accesses the Unity engine.

[0030] Beneficial effects:

[0031] The present invention can dynamically change the image deformation according to the user's position, and can limit the influence area of the deformation effect, achieving the effect of scaling and natural transition of the texture in a local area. The implementation method of the present invention is simple and can improve the user experience. Description of the drawings

[0032] Figure 1 It is the interface diagram of the newly created Render Texture;

[0033] Figure 2 It is the environmental parameter setting interface of the newly created material ball;

[0034] Figure 3 It is the setting interface of the new Shader shader;

[0035] Figure 4 It is the interface for displaying the assignment of the material texture;

[0036] Figure 5 It is the interface display of the Shader shader after the program is set;

[0037] Figure 6 What is shown is the VR video image content projected in the form of a skybox in the state without position offset;

[0038] Figure 7 What is shown is the deformation situation of the image after the user's position moves to the left. Detailed implementation manners

[0039] Virtual reality live-action videos need to be played on professional VR devices. Therefore, the production of the videos also needs to be completed in the corresponding VR development engine. The present invention uses the Unity engine as the implementation platform, and the present invention will be further described in detail below.

[0040] The method for dynamically changing the image deformation according to the user's position in the virtual reality live-action video in this embodiment includes the following steps:

[0041] Step 1) Project the images in the live-action video onto the skybox of the spherical structure

[0042] 1.1) Streaming media management

[0043] In Unity, set the management method of the video to streaming media loading, then set the streaming media storage path in the engine, import the virtual reality live-action video file into the storage path, note that the path name needs to meet the built-in access requirements of Unity, and store it under the Streaming Asset folder. Since videos are usually imported in the form of material resources in Unity, the streaming media video loading method needs to provide code to access a specific video storage path and convert the video format into a general format recognized by the engine through code. The core code is as follows.

[0044] path = Application.streamingAssetsPath + " / Video Name.mp4";

[0045] vp = GameObject.Find("GameObject").GetComponent<VideoPlayer>();

[0046] vp.url = " file: / / / " + path;

[0047] vp.Play();

[0048] 1.2) Video rendering material conversion

[0049] The video playback function needs to be implemented through the video playback component in Unity. Therefore, first create a virtual object for playing the video, and then add the video playback component to this virtual object. It should be noted here that in the video projection mode without a screen, the display of the picture is presented through the render texture Render Texture. Therefore, a new Render Texture needs to be created and assigned to the video playback component, and the specific parameters are modified according to the video information, such as Figure 1 as shown.

[0050] 1.3) VR environment mapping

[0051] The purpose of this step is to present the video in a panoramic view. In this embodiment, the implementation method adopted is skybox mapping. Therefore, it is necessary to connect the video to the skybox. The steps are as follows:

[0052] 1) Create a new VR environment material sphere in the engine to replace the default skybox material sphere, and the parameter settings are as Figure 2 shown.

[0053] 2) Modify the Shader shader in the default material sphere to the Shader shader that can be used for the new skybox. It should be noted here that the new Shader shader needs to inherit the source code of the default skybox shader, as Figure 3 shown.

[0054] 3) Assign the previously created render texture Render Texture to the spherical HDR map area (Spherical HDR) in the new material sphere, so that the material texture information of the video can be displayed in the new material sphere, as Figure 4 shown.

[0055] Step 2) Dynamically change the image deformation according to the user's position

[0056] In this embodiment, it is achieved by modifying the image rendering information in the Shader shader according to the user's position information. The specific steps are as follows:

[0057] 1) Connect the VR device in the engine (import the plugin for connecting the VR device), and require the user to set the headset height and the operation area radius, and the operation area radius is used to limit the moving range of the user in the VR scene.

[0058] 2) Obtain the initial position of the user through the code (the default position of the user when the VR program starts running, that is, the coordinate origin position (0, 0, 0)), and monitor the position change of the user in real time during the running of the program, and compare this position with the initial position to obtain the moving direction and distance, etc. (the height difference between the VR headset and the reference plane comes from the height set by the user when initializing the VR device), and map this direction vector to the texture space. The core code is as follows.

[0059]

[0060] The Start method is a method in Unity that is executed once at the start of the game scene. Here, it assigns the initial position of the VR_HMD object (obtaining its coordinate position in three-dimensional space through transform.position) to the User_position variable, thereby recording the initial position of the user (represented by the VR headset) as a benchmark for subsequent operations such as comparing position changes.

[0061]

[0062] The Update method is executed in each frame and is used to update relevant data and logic in real time. The real-time position of the VR_HMD in the current frame is obtained through the code "dynamic_position = VR_HMD.transform.position" and assigned to the dynamic_position variable, so that the latest position of the VR headset can be tracked at all times.

[0063] "User_direction1 = dynamic_position - User_position" calculates the direction vector pointing from the user's initial position to the current position, which reflects the direction of the user's movement. And "User_direction2 = User_position - dynamic_position" calculates the vector in the opposite direction. These two direction vectors will be used later to set the scaling center points in different directions.

[0064] "distance1 = Vector3.Distance(Vector3.zero, dynamic_position)" calculates the distance from the current position of the VR_HMD to the origin of the coordinate system. This distance value distance1 can be used to measure how far the user is from the origin of the scene and will be used later to determine the degree of scaling.

[0065] "x = (Mathf.Atan2(User_direction1.z, User_direction1.x) + Mathf.PI) / (2 * Mathf.PI)" uses the Atan2 function to calculate an angle value based on the components of the User_direction1 vector in the x and z directions, then adds Mathf.PI (here its value is π) for angle offset adjustment, and finally divides by 2 * Mathf.PI to normalize the angle value to the range of 0 to 1, and the obtained value is assigned to the x variable.

[0066] "y = Mathf.Acos(User_direction1.y) / Mathf.PI" uses the Acos function to calculate an angle value based on the component of the User_direction1 vector in the y direction, and then divides it by Mathf.PI for normalization to make its range between 0 and 1, and assigns the result to the y variable.

[0067] The above conversion maps the direction vector User_direction1 to a certain position in the texture space, which is usually represented in the form of UV coordinates. Its UV coordinates in this space are (x, y). This position is the scaling center point in the approaching direction.

[0068] 3) Connect information such as the corresponding scaling center point and scaling degree into the Shader properties of the material ball m in the code. The core code is as follows.

[0069] m.SetVector("_MinifyCenter1", new Vector4(x, y, 0, 0)); / / Set the mapping coordinates in the approaching direction

[0070] m.SetVector("_MinifyCenter2", new Vector4(Mathf.Abs(1 - x), y, 0, 0)); / / Set the mapping coordinates in the away direction

[0071] m.SetFloat("_MinifyAmount1", distancel / 20); / / Set the ratio of the approaching distance

[0072] m.SetFloat("_Radius", (m.GetFloat("_MinnifyAmount1") * 0.25f)); / / Set the deformation radius

[0073] m.SetFloat("_Falloff", m.GetFloat("_MinifyAmountl")); / / Set the attenuation radius

[0074] The first two lines of code are used to pass two four-dimensional vector data to the Shader shader of the material ball m.

[0075] _MinifyCenter1 and _MinifyCenter2 are the names of vector variables in the Shader shader. These two vectors are used to represent the scaling center point coordinates in the approaching and away directions respectively (only the first two dimensions are used here, that is, the coordinates on the xy plane, and the last two dimensions are set to 0). The abscissa of _MinifyCenter2 uses mathf.abs(1

[0076] -x), the mathf.abs function is used to obtain the absolute value of 1 - x. This means that the abscissa of _Minifycenter2 is symmetric about the coordinate origin with respect to the abscissa x of _Minifycenter1. In this way, it is more convenient to set the vector information related to the center point in the Shader shader for the far - away direction.

[0077] "m.Setfloat("_MinifyAmount1", distance1 / 20)" passes the value obtained by dividing distance1 by 20 (where "20" is the set value of the operation radius) to the _MinifyAmount1 variable, which is used to set the first coefficient related to the scaling degree and so on.

[0078] "m.Setfloat("_Radius", (m.getfloat("_MinifyAmount1") * 0.25f)" sets the _Radius variable according to the value of _MinifyAmount1 obtained multiplied by 0.25f (f indicates that this is a single - precision floating - point number). The _Radius variable is used to represent a range radius that defines the influence area of the graphic effect (scaling effect). 0.25 is a limiting coefficient designed to confine the deformation in the near and far directions within their respective half - sphere ranges to avoid excessive deformation.

[0079] "m.Setfloat("_Falloff", m.getfloat("_MinifyAmount1"))" sets the _Falloff variable with the value of _MinifyAmount1 obtained. The _Falloff variable is used to control the attenuation of the graphic effect at the boundary. In this embodiment, the value of _MinifyAmount1 is directly assigned to it without multiplying by a limiting coefficient like the _Radius variable, aiming to make the deformation effect transition more natural in combination with the subsequent smoothing function.

[0080] The rendering effect in the Shader shader of the material ball m after setting is as Figure 5 shown ( Figure 5 the specific numerical values of the parameters in

[0081] 4) Establish a comparison vector i.dir, which represents the direction vector from the user's initial position to a certain pixel position. Then, similar to the operation on the direction vector User_direction1, map it to the position uv in the texture space. The core code is as follows.

[0082] float theta = atan2(i.dir.z, i.dir.x) + UNITY_PI;

[0083] float phi = acos(i.dir.y);

[0084] uv = float2(theta / UNITY_TWO_PI, phi / UNITY_PI);

[0085] 5) Adjust the position of the uv coordinates relative to the scaling center according to the scaling factor to produce a scaling effect

[0086] The calculations here are implemented in the Shader shader. Since it is more convenient to implement through it, the positions in the code are matched into the Shader shader in step 3) above.

[0087] The core code of the calculation process is as follows.

[0088]

[0089] "float2 center = _MinifyCenter1.xy" extracts the first two dimensions from the previously set four-dimensional vector _MinifyCenter1 and assigns them to the center variable to obtain the corresponding center point coordinates on the UV coordinate plane.

[0090] "float2 delta = uv - center" calculates the difference vector delta between the uv coordinates corresponding to the vector i.dir and center, which is used to measure its offset relative to center.

[0091] "float dist = length(delta)" calculates the length of the difference vector delta, that is, obtains the distance dist between the uv coordinates corresponding to the comparison vector i.dir and the center point center. Then, through the condition "if(dist < _Radius)", it is judged whether this distance is less than the set range radius _Radius. If it is less, the above uv coordinates (corresponding to the comparison vector i.dir) are adjusted to achieve the corresponding graphic effect.

[0092] Function of the smoothstep function: The factor variable is calculated by "smoothstep(_Radius, _Radius - _Falloff, dist)". The smoothstep function returns a smooth transition value between 0 and 1 based on the value of dist within the range of _Radius and _Radius - _Falloff. When dist approaches _Radius, factor approaches 0; when dist approaches _Radius - Falloff, factor approaches 1. However, since dist represents distance and cannot be less than 0, the actual value range of the factor variable is only 0 - 0.25.

[0093] Application of the lerp function: "lerp(1.0, 1 + _MinifyAmount1, factor)" performs linear interpolation between 1.0 and 1 + _MinifyAmount1 according to the value of factor. When factor is 0, the interpolation result is 1.0, and at this time, the offset of the part "delta * lerp(1.0, 1 + _MinifyAmount1, factor)" to center is relatively small; when factor is 1, the interpolation result is 1 + _MinifyAmount1. "delta * lerp(1.0, 1 + _MinifyAmount, factor)" scales the difference vector delta according to this scaling factor determined by the lerp function. Since delta represents the relative offset starting from the center of center, scaling it is equivalent to changing this offset, thereby changing the position of the pixel uv coordinate relative to the scaling center of center and generating a scaling effect. As can be seen from the above, the actual value range of the factor variable is only 0 - 0.25, so it is equivalent to restricting the magnitude of this scaling factor determined by the lerp function, slowing down the overall scaling intensity.

[0094] For _MinifyAmount2, the processing method is similar, but the effect is opposite. The core code is as follows.

[0095]

[0096] Generally speaking, the above code processes the uv coordinates of pixels around parameters such as the center point coordinates and range radius through a series of parameter settings, distance calculations, conditional judgments, and interpolation operations. The uv coordinates within the _Radius range will produce a scaling effect according to the combined action of factor and _MinifyAmount1 and 2, while the uv coordinates outside the range remain unchanged, thus achieving texture scaling and natural transition within a local area.

[0097] The comparison of the effects after the final program runs is as follows:

[0098] Figure 6 The displayed VR video image content is projected in the form of a skybox without position offset. Figure 7 The displayed image deformation after the user's position moves to the left can be seen. At this time, the left skybox screen shows an enlarged effect with the moving direction as the deformation center. Correspondingly, the right skybox screen shows a reduced effect with the opposite moving direction as the deformation center.

[0099] In the present invention, the acquisition of the viewer's position can be generated by the positioning device incorporated in the virtual reality device. For virtual reality devices without a positioning function, the processing method in the above embodiments can also be referred to, and the usage height can be adjusted according to the initialization settings of the device.

[0100] In the present invention, setting the change ratio according to the range of the operation area can limit the user experience within a safe range.

Claims

1. A method for dynamically changing image deformation according to user position in a virtual reality live video, which uses the Unity engine as an implementation platform, and is characterized in that it includes the following steps: Step 1) Projecting the image in the real-shot video into the sky box of the spherical structure; Step 2) Modify the image rendering information in the Unity engine Shader according to the user's position to achieve dynamic image deformation according to the user's position.

2. The method for dynamically changing image deformation according to user position in a virtual reality live video according to claim 1, characterized in that step 2) is specifically implemented as follows: a) Get the user's initial position and its dynamic position in real time, compare the two positions, get the user's moving direction User_direction1 and distance distance1, and correspond the direction vector User_direction1 to the texture space, get its position coordinates (x, y) in the texture space, and use them as the scaling center point coordinates of the user's approach direction; b) In the Shader shader, two vector variables _MinifyCenter1 and _MinifyCenter2 are used to carry the coordinates of the scaling center point in the direction of the user approaching and moving away, respectively, and the scaling factor variable MinifyAmount1 is set to distance1 / R, the deformation radius _Radius is _MinifyAmount1*0.25, and the attenuation radius _Falloff is _MinifyAmount1, where: R represents the set operating area radius, which is used to limit the user's movement range; c) Establish a comparison vector i.dir, which represents the direction vector from the user's initial position to a certain pixel position, and also correspond the direction vector to the texture space, and its position is represented as uv; d) According to the scaling factors _MinifyAmount1 and _MinifyAmount2, the position of the uv coordinate relative to the scaling center is adjusted to produce a scaling effect Amplification effect: Assign the horizontal and vertical coordinates of _MinifyCenter1 to the variable center, set the difference vector delta = uv-center, and calculate the length of delta dist; When dist<_Radius, perform the following operations: Through the smoothstep function, according to the value of dist in the interval of _Radius and _Radius-_Falloff, a smooth transition value factor between 0 and 1 is returned; Use the lerp function to perform linear interpolation between 1.0 and 1+_MinifyAmount1 according to the value of factor, and then scale the difference vector delta according to the scaling factor determined by the lerp function to obtain the new uv coordinate relative to the center; Zoom out effect: Let _MinifyAmount2=-_MinifyAmount1; Assign the horizontal and vertical coordinates of _MinifyCenter2 to the variable center, set the difference vector delta = uv-center, and calculate the length of delta dist; When dist<_Radius, perform the following operations: Through the smoothstep function, according to the value of dist in the interval of _Radius and _Radius-_Falloff, a smooth transition value factor between 0 and 1 is returned; The lerp function is used to perform linear interpolation between 1.0 and 1+_MinifyAmount2 according to the value of factor, and then the difference vector delta is scaled according to the scaling factor determined by the lerp function to obtain the new uv coordinate relative to the center.

3. The method for dynamically changing image deformation according to user position in virtual reality live video according to claim 2 is characterized in that, in step b), the horizontal coordinate in _MinifyCenter1 is x, and the vertical coordinate is y, and the horizontal coordinate and vertical coordinate in _MinifyCenter2 are set to mathf.abs(1-x) and y respectively according to the horizontal coordinate and vertical coordinate in _MinifyCenter1, and the mathf.abs function is used to obtain the absolute value of 1-x.

4. The method for dynamically changing image deformation according to user position in a virtual reality live video according to claim 2 is characterized in that the position information used to represent the height of the VR head display in the dynamic position is generated by user settings when the VR device is connected to the Unity engine, or is generated by a positioning device connected to the virtual reality device.

5. The method for dynamically changing image deformation according to user position in a virtual reality live video according to claim 2 is characterized in that the radius of the operating area is determined by user settings when the VR device is connected to the Unity engine.

Images