Method and device for playing VR (Virtual Reality) video on augmented reality equipment

Abstract

The invention relates to a method and device for playing a VR video on an augmented reality device, and the method comprises the steps: responding to a VR video played by a user, and opening a video playing window in an augmented reality space of the augmented reality device; acquiring a VR video image to be played, and sampling and rendering the VR video image according to the sight line of human eyes and the position of the video playing window in the augmented reality space to obtain a target image; the VR video comprises a 180-degree video or a 360-degree video; and displaying the target image in the video playing window. The technical scheme provided by the invention provides a method for providing a VR video playing function in a shared space mode. By adopting the technical scheme provided by the invention, the purpose that the video playing window displays different image contents along with different visual angles of human eyes can be achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of extended reality technology, and in particular to a method and apparatus for playing VR videos on an extended reality device. Background Art

[0002] VR video is a type of video shot based on virtual reality technology using a dedicated 360° or 180° panoramic camera. Its core feature is that it can capture all-round image information and provide viewers with a full range of vision. When viewers watch it, they can break through the constraints of the fixed perspective of traditional videos. For example, when watching a VR video of a concert, the audience puts on a VR headset and feels as if they are in the center of the concert. By turning their heads, they can see the performances from all angles on the stage and the reactions of the audience below.

[0003] Shared Space is an experience mode provided by extended reality devices. In this experience mode, users can experience the fusion of the physical world and virtual content at the same time. This experience mode allows users to interact with the surrounding environment and other people without being completely isolated in a virtual environment. Shared Space is very suitable for situations that require attention to both the real world and digital content, such as collaborative work, social interaction, and enhanced real-time applications.

[0004] In the shared space mode, how to provide VR video playback and presentation is an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method and apparatus for playing VR videos on an extended reality device.

[0006] In a first aspect, the present disclosure provides a method for playing a VR video on an extended reality device, the method comprising:

[0007] In response to the user playing the VR video, opening a video playback window in an extended reality space of the extended reality device;

[0008] Acquire a VR video image to be played, and sample and render the VR video image according to the line of sight of human eyes and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes a 180° or 360° video;

[0009] The target image is displayed in the video playback window.

[0010] In a second aspect, the present disclosure further provides a device for playing a VR video on an extended reality device, the device comprising:

[0011] A first display module, configured to open a video playback window in an extended reality space of an extended reality device in response to a user playing a VR video;

[0012] A production module, used to obtain a VR video image to be played, and sample and render the VR video image according to the line of sight of human eyes and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes a 180° or 360° video;

[0013] The second display module is used to display the target image in the video playback window.

[0014] In a third aspect, the present disclosure further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] A storage device for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for playing VR videos on an extended reality device as described above.

[0018] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for playing VR videos on an extended reality device as described above.

[0019] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0020] The technical solution provided by the disclosed embodiment opens a video playback window in the extended reality space of the extended reality device in response to the user playing the VR video; obtains the VR video image to be played, and samples and renders the VR video image according to the human eye line of sight and the position of the video playback window in the extended reality space to obtain the target image; the VR video includes 180° or 360° video; and displays the target image in the video playback window. Using the technical solution provided by the present application, the VR video is displayed in the extended reality space with the help of the video playback window, instead of using the entire environmental image to display the VR video. In this way, other information can also be displayed with the help of other windows or environmental images, thereby achieving the purpose of the shared space mode. In other words, the technical solution provided by the present application provides a method for providing a VR video playback function in the shared space mode. Moreover, the technical solution provided by the present application can achieve the purpose of displaying different image contents in the video playback window according to different perspectives of the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A flowchart of a method for playing a VR video on an extended reality device provided in an embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of an extended reality space provided by an embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of scaling a video playback window provided in an embodiment of the present disclosure;

[0026] Figure 4 Schematic diagram of a structure of an apparatus for playing VR videos on an extended reality device in an embodiment of the present disclosure;

[0027] Figure 5 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] Figure 1 A flowchart of a method for playing VR videos on an extended reality device is provided in an embodiment of the present disclosure. The method can be executed by an apparatus for playing VR videos on an extended reality device. The apparatus can be implemented in software and / or hardware, and the apparatus can be configured in the extended reality device.

[0031] The extended reality devices described in this application may include but are not limited to the following types:

[0032] The computer-side extended reality device uses the PC to perform related calculations and data output of the extended reality function, and the external computer-side extended reality device uses the data output by the PC to achieve the extended reality effect.

[0033] Mobile extended reality devices support the setting of mobile terminals (such as smart phones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations of the extended reality function and outputs data to the mobile extended reality device, such as watching extended reality videos through the mobile terminal's APP.

[0034] The all-in-one extended reality device has a processor for performing relevant calculations of virtual functions, and thus has independent extended reality input and output functions. It does not need to be connected to a PC or mobile terminal and has a high degree of freedom in use.

[0035] See also Figure 1 The method for playing a VR video on an extended reality device provided by the present application may include:

[0036] S1. In response to a user playing a VR video, a video playback window is opened in an extended reality space of an extended reality device.

[0037] The extended reality space is a virtual scene created with the help of environmental images when the extended reality device is running. The virtual scene can be a simulation of the real world, a semi-simulated and semi-fictional virtual scene, or a purely fictional virtual scene. The extended reality space can include the sky, land, ocean, etc. The land can include environmental elements such as deserts and cities, and users can move in the extended reality space. The environmental image is equivalent to the desktop wallpaper displayed after the computer is turned on.

[0038] The video playback window may be, for example, an area in the extended reality space for playing VR videos at 360°, 180°, or other angles. In practice, the video playback window may be, for example, an area in a control panel for playing VR videos.

[0039] VR video has a corresponding spherical model. The spherical model can be, for example, a three-dimensional geometric model for carrying VR video images, which is an idealized sphere, and its surface is used as a mapping carrier for the VR video image content. There is a correspondence between the points on the surface of the spherical model and the pixel points in the VR video image, and this correspondence can be determined with the help of spherical projection methods such as equirectangular projection or AEP (Adjusted equal-area projection). With the correspondence between the points on the surface of the spherical model and the pixel points in the VR video image, the VR video image can be accurately "attached" to the surface of the spherical model. It should also be noted that if a Cartesian coordinate system is constructed in the extended reality space, the center and radius of the spherical model are determined in the construction of the Cartesian coordinate system.

[0040] S2. Obtain the VR video image to be played, sample and render the VR video image according to the human eye line of sight and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes 180° or 360° video.

[0041] VR video images refer to image frames in VR videos.

[0042] In an extended reality device, a variety of sensors are usually provided, such as an IMU (Inertial Measurement Unit), a 3Dof & 6Dof (three degrees of freedom & six degrees of freedom) sensor, and an eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera). With the help of the data collected by these sensors, the line of sight of the human eye can be determined.

[0043] The position of the video playback window in the extended reality space can be obtained based on the running data of the application used to play the VR video.

[0044] The essence of this step is to select pixels that can be seen by the user from the VR video image based on the line of sight of the human eye and the position of the video playback window in the extended reality space, and to produce a target image that needs to be displayed in the video playback window.

[0045] S3. Display the target image in the video playback window.

[0046] Figure 2 A schematic diagram of an extended reality space provided by this application. Figure 2 , the extended reality space includes the user ( Figure 2 D1 and D2 represent two possible positions of the user in the extended reality space), video playback windows A1A2A3A4 and a ball model corresponding to the VR video. Figure 2In the figure, ring L1 and ring L2 are two rings on the spherical model corresponding to the VR video.

[0047] By adopting the technical solution provided by the present application, from the perspective of user experience, when the user is in the extended reality space, the video playback window A1A2A3A4 is similar to a window, and the user can watch the VR video image "attached" to the ball model through the video playback window A1A2A3A4. Moreover, due to the limitation of the video playback window A1A2A3A4, the user can only see the image of the local area "attached" to the surface of the ball model. For example, when the user is in Figure 2 When the user is at position D1, only the image "attached" to the surface of the ball model in the area B1B2B3B4 can be seen. Figure 2 When the user is at position D2, only the image "attached" to the surface of the ball model in area C1C2C3C4 can be seen. Figure 2 When the user is at position D1 in the video playback window, the target image displayed in the video playback window is produced based on the pixel points in the VR video image corresponding to the points in the area B1B2B3B4 on the surface of the ball model. Figure 2 When the position D2 is reached, the target image displayed in the video playback window is produced based on the pixel points in the VR video image corresponding to the points in the C1C2C3C4 area on the surface of the ball model.

[0048] It should be emphasized that in the extended reality space, the ball model is invisible to the user. Moreover, in the process of implementing the technical solution provided by this application, it is not necessary to attach the VR video image to the ball model.

[0049] The technical solution provided by the present application opens a video playback window in the extended reality space of the extended reality device in response to the user playing a VR video; obtains the VR video image to be played, and samples and renders the VR video image according to the line of sight of the human eye and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes a 180° or 360° video; and the target image is displayed in the video playback window. Using the technical solution provided by the present application, the VR video is displayed in the extended reality space with the help of a video playback window, rather than using the entire environmental image to display the VR video. In this way, other information can also be displayed with the help of other windows or environmental images, thereby achieving the purpose of the shared space mode. The technical solution provided by the present application can achieve the purpose of displaying different image contents in the video playback window according to different perspectives of the human eye, so that the user has the feeling of "looking through" the window to observe the scenery outside the window.

[0050] In the above technical solution, there are many specific implementation methods for "sampling and rendering the VR video image according to the line of sight of the human eye and the position of the video playback window in the extended reality space to obtain the target image", and this application does not limit this. In some embodiments, optionally, the VR video image is a UV image; "sampling and rendering the VR video image according to the line of sight of the human eye and the position of the video playback window in the extended reality space to obtain the target image" may include: determining the intersection of the line of sight of the human eye and the ball model according to the position of the human eye in the extended reality space, the line of sight of the human eye, the position of the video playback window in the extended reality space, and the position of the ball model in the extended reality space; based on the intersection of the line of sight of the human eye and the ball model, sampling the VR video image to obtain the sampling result; rendering based on the sampling result to obtain the target image.

[0051] UV image is a coordinate representation used in three-dimensional graphics processing, especially in the field of VR video images. U and V can be compared to the X and Y coordinate axes on a two-dimensional plane. When a two-dimensional image (such as a frame in a VR video) needs to be accurately "attached" to the surface of the model, it is done with the help of UV coordinates. The U coordinate is used to control the horizontal stretching, tiling or alignment of the image; the V coordinate is used to control the vertical stretching, tiling or alignment of the image. In the process of VR video production, in order to make the VR video image wrap the ball model, it is necessary to determine the UV value corresponding to each point on the surface of the ball model corresponding to the VR video image.

[0052] The position of the video playback window in the extended reality space and the position of the ball model in the extended reality space can be obtained by reading the running data of the application for playing VR videos in the extended reality device.

[0053] The position of the human eye in the extended reality space and the direction of the human eye's line of sight can be calculated based on the data collected by the sensors of the extended reality device.

[0054] The intersection of the human eye's line of sight and the ball model means that if the ball model is visible to the user, the human eye can observe the position of the intersection on the ball model, which also means that the pixel point corresponding to the intersection in the VR video image is visible to the human eye.

[0055] The intersection of the human eye's line of sight and the ball model is a point on the ball model surface. When "sampling the VR video image based on the intersection of the human eye's line of sight and the ball model", the pixel point in the VR video image corresponding to the intersection can be determined and extracted by querying the correspondence between each point on the ball model surface and the pixel point in the VR video image, thereby achieving the purpose of sampling the VR video image to obtain the sampling result.

[0056] The technical solution provided by this application can realize the purpose of displaying different image contents in the video playback window according to different viewing angles of the human eye, so that the user has the feeling of "looking through" the window to observe the scenery outside the window. In order to achieve this effect, in practice, the Portal rendering method can also be used for rendering. However, this rendering method needs to rely on Stencil Test to be implemented. And Stencil Test requires combination operations between multiple models. The combination of multiple models involves a lot of calculations and data processing, including the calculation of positional relationships between models, the judgment of occlusion relationships, the determination of rendering order, etc. These complex calculations and processing processes will take up a lot of system resources, such as the computing power of the CPU and GPU, memory bandwidth, etc., which will lead to a decline in the performance of electronic equipment, such as screen freezes, reduced frame rates, response delays and other performance issues.

[0057] By setting the position of the human eye in the extended reality space, the direction of the human eye's sight, the position of the video playback window in the extended reality space, and the position of the ball model in the extended reality space, the intersection of the human eye's sight and the ball model is determined; based on the intersection of the human eye's sight and the ball model, the VR video image is sampled to obtain the sampling result. Since the Portal rendering method is not required, the performance problems caused by the Portal rendering method can be avoided.

[0058] On the basis of the above technical solution, optionally, "determining the intersection of the line of sight of the human eye and the ball model according to the position of the human eye in the extended reality space, the line of sight direction of the human eye, the position of the video playback window in the extended reality space, and the position of the ball model in the extended reality space" may include: determining the line of sight beam according to the position of the human eye in the extended reality space and the line of sight direction of the human eye; the line of sight beam includes multiple lines of sight; based on the position of the video playback window in the extended reality space, determining the target line of sight among the multiple lines of sight; the target line of sight is a line of sight that can pass through the video playback window; determining the intersection of the target line of sight and the ball model; "based on the intersection of the line of sight of the human eye and the ball model, sampling the VR video image to obtain a sampling result" may include: based on the intersection of the target line of sight and the ball model, sampling the VR video image to obtain a sampling result.

[0059] When the human eye watches a VR video image, it emits a line of sight. The line of sight includes multiple lines of sight, some of which intersect with the border of the video playback window or other non-VR video image objects. These lines of sight do not fall on the ball model used to present the VR video, so they are not helpful for watching the VR video. Another part of the line of sight can pass through the video playback window and intersect with the ball model. Only through these lines of sight that intersect with the ball model can the human eye observe the content presented by the VR video image. The target line of sight is the line of sight that can pass through the video playback window and intersect with the ball model. By setting the position of the video playback window in the extended reality space, the target line of sight is determined among multiple lines of sight. In essence, the line of sight is screened before determining the intersection point of the human eye's line of sight with the ball model. The purpose of this setting is to reduce the computational complexity of determining the intersection point of the human eye's line of sight with the ball model.

[0060] Further, determining the intersection of the target sight line and the ball model may include: obtaining the coordinate value of the intersection of the target sight line and the ball model in the Cartesian coordinate system based on the human eye position, the sight direction of the target sight line, and the spherical equation of the ball model in the Cartesian coordinate system; converting the coordinate value of the intersection of the target sight line and the ball model in the Cartesian coordinate system to the coordinate value of the intersection of the target sight line and the ball model in the spherical coordinate system; converting the coordinate value of the intersection of the target sight line and the ball model in the spherical coordinate system to the coordinate value of the intersection of the target sight line and the ball model in the UV coordinate system. S130 may include: sampling the VR video image based on the coordinate value of the intersection of the target sight line and the ball model in the UV coordinate system to obtain a sampling result.

[0061] For example, assuming that in a Cartesian coordinate system, the center of the spherical model is C(x c ,y c , z c ), with a radius of r. The starting point of the target line of sight P(x p ,y p , z p ), the direction is The intersection point between the target line of sight and the sphere is assumed to be T(x t ,y t , z t ).but

[0062] (x t ,y t , z t )=(x p ,y p , z p )+t·(d x , d y , d z ) (1)

[0063] The spherical equation of the sphere model is

[0064] (xx c ) 2 +(yy c ) 2 +(zz c ) 2 =r 2 (2)

[0065] Substituting equation (1) into equation (2), we can obtain the quadratic equation about t: By solving the quadratic equation, we can get t. Substituting t into equation (1), we can get the coordinate value of the intersection point of the target line of sight and the spherical model in the Cartesian coordinate system.

[0066] X=cos(θ)cos(φ) (3)

[0067] Y=sin(θ)(4)

[0068] Z=--cos(θ)sin(φ) (5)

[0069] φ=tan -1 (-Z / X) (6)

[0070]

[0071] U=(φ+π) / 27π=φ / 2π+0.5 (8)

[0072] V=(θ+π / 2) / π=θ / π+0.5 (9)

[0073] Subsequently, the coordinate value of the intersection point of the target line of sight and the ball model in the Cartesian coordinate system can be converted into the coordinate value of the intersection point of the target line of sight and the ball model in the spherical coordinate system with the help of formula (3)-formula (7); the coordinate value of the intersection point of the target line of sight and the ball model in the spherical coordinate system can be converted into the coordinate value of the intersection point of the target line of sight and the ball model in the UV coordinate system with the help of formula (8)-formula (9).

[0074] On the basis of the above technical solution, optionally, the method may also include: in response to a zooming operation on the video playback window, adjusting the size of the video playback window and the position and size of the ball model, so that when the relative position of the human eye and the video playback window remains unchanged, the VR video image content observed through the video playback window is the same before and after the zooming operation on the video playback window.

[0075] The zooming operation on the video playback window may be, for example, an operation indicating that the user wishes to zoom in or zoom out the video playback window. Zooming in the video playback window means increasing the size of the video playback window; zooming out the video playback window means reducing the size of the video playback window.

[0076] In practice, the content displayed in the video playback window is a local area in the VR video image. "When the relative position between the human eye and the video playback window remains unchanged, the VR video image content observed through the video playback window is the same before and after the video playback window is zoomed in and out." For example, before and after the video playback window is zoomed in and out, the window always displays the image of the same specific area of ​​the VR video image.

[0077] For example, in Figure 3 In the figure, before the video playback window is enlarged, the video playback window is MN and the ball model used is S; after the video playback window is enlarged, the video playback window is QP and the ball model used is T. Point A represents the user. Figure 3 Before the video playback window is enlarged, in the extended reality space, the user can see the image of the E1F1 segment of the ball model S "attached" through the video playback window MN. After the video playback window is enlarged, in the extended reality space, the user can see the image of the E2F2 segment of the ball model T "attached" through the video playback window QP. And the image of the E1F1 segment of the ball model S "attached" and the image of the E2F2 segment of the ball model T "attached" are the same image.

[0078] When the video playback window is enlarged, the image originally displayed in the video playback window will be enlarged accordingly; and when the video playback window is reduced, the image originally displayed in the video playback window will be reduced accordingly. From the user's experience point of view, there is a linkage relationship between the video playback window and the displayed image, and the image will change size in real time as the window is scaled.

[0079] In response to the zooming operation on the video playback window, the position and size of the ball model need to be adjusted based on the size of the video playback window. This is because, when the relative position between the human eye and the video playback window remains unchanged, the size change of the video playback window will cause the maximum viewing angle of the human eye limited by the video playback window to change. Figure 3In the figure, before the video playback window is zoomed in, the user is located at the best viewing point relative to the video playback window. Specifically, the user is located on the perpendicular bisector AC of the video playback window, and the user is located at the center of the spherical model S. That is, AC is perpendicular to MN, C is the midpoint of MN, and A is the center of the spherical model S. At this time, the maximum viewing angle limited by the video playback window MN is γ. After the video playback window is zoomed in, the maximum viewing angle of the human eye limited by the video playback window QP is γ+β. The intersection point E1 of the line of sight AM and the circle S and the intersection point E2 of the line of sight AQ and the circle T need to correspond to the same UV coordinate of the VR video image. The purpose of adjusting the position and size of the spherical model is to make point E1 on circle S and point E2 on circle T correspond to the same UV coordinate of the VR video image.

[0080] Furthermore, in response to a zooming operation on the video playback window, the size of the video playback window and the position and size of the ball model are adjusted, including: in response to a zooming operation on the video playback window, determining a zoom factor of the video playback window; based on the zoom factor of the video playback window, determining a target size of the video playback window and a zoom factor of the ball model; based on the zoom factor of the ball model, determining a target radius and a target center position of the ball model; adjusting the size of the video playback window to the target size; adjusting the radius of the ball model to the target radius, and adjusting the center of the ball model to the target center position.

[0081] The scaling factor of the video playback window may be, for example, a ratio of the size of the video playback window after the scaling operation to the size before the scaling operation. In practice, it may be determined according to a specific scaling operation on the video playback window. For example, if a user scales the video playback window by dragging the video playback window border, the scaling factor of the video playback window may be determined by the distance the user drags the video playback window border to move.

[0082] For example, if before the scaling operation on the video playback window, the size of the video playback window is m, the radius of the sphere model is r, the scaling factor of the video playback window is psacle, and the scaling factor of the sphere model is rscale, then the target size of the video playback window is psacle·m, and the target radius of the sphere model is rscale·r.

[0083] There are many specific implementation methods for "determining the scaling factor of the ball model based on the scaling factor of the video playback window", and this application does not limit this. In practice, a functional relationship between the scaling factor of the video playback window and the scaling factor of the ball model can be pre-constructed. Based on the functional relationship and the scaling factor of the video playback window, the scaling factor of the ball model is determined.

[0084] Optionally, determining the scaling factor of the ball model based on the scaling factor of the video playback window may include: determining a first viewing angle, the first viewing angle being the maximum viewing angle of the human eye limited by the video playback window before adjusting the size of the video playback window; determining the scaling factor of the ball model based on the first viewing angle and the scaling factor of the video playback window.

[0085] Continue to see Figure 3 In order to make the intersection point E1 between the line of sight AM and the circle S and the intersection point E2 between the line of sight AQ and the circle T correspond to the same UV coordinate of the VR video image, the center of circle T can be set to B, and BE2 can be parallel to AE1. And the spherical model after the video playback window scaling operation is set to be inscribed with the model before the video playback window scaling operation, and the inscribed point G and the user's position (point A) are separated on both sides of the video playback window MN, and the inscribed point G is located on the perpendicular bisector AC of the video playback window. The radius of the spherical model before the scaling operation of the video playback window is r, and the target radius of the spherical model after the scaling operation of the video playback window is R. The scaling coefficient of the spherical model is rscale, E2D is perpendicular to BD, and under the above-mentioned limited conditions,

[0086]

[0087] Based on equation (11) and equation (12), we can get equation (13):

[0088]

[0089] According to formula (13), based on the first viewing angle γ and the scaling factor psacle of the video playback window, the scaling factor rscale of the ball model can be obtained. Based on the scaling factor rscale of the ball model, the center position and radius of the circle T can be obtained.

[0090] Further, determining the target radius and target center position of the spherical model based on the scaling coefficient of the spherical model may include: determining the target radius of the spherical model based on the initial radius of the spherical model and the scaling coefficient of the spherical model; determining the target center position of the spherical model based on the initial radius of the spherical model, the initial center position of the spherical model and the target radius of the spherical model.

[0091] For example, in Figure 3In the example in , since the center B of the ball model after the video playback window is scaled and the center A of the ball model before the video playback window is scaled are both located on the perpendicular bisector AC of the video playback window, and the ball model after the video playback window is scaled and the ball model before the video playback window is scaled are inscribed at point G, and the inscribed point G is located on the perpendicular bisector AC of the video playback window, BA=Rr can be obtained, where R is the target radius of the ball model and r is the initial radius of the ball model. Then the position coordinates of the target center B of the ball model can be obtained.

[0092] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0093] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0094] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0095] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0096] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0097] Figure 4 Schematic diagram of the structure of a device for playing VR videos on an extended reality device in an embodiment of the present disclosure. The device for playing VR videos on an extended reality device provided in an embodiment of the present disclosure is applicable to the extended reality device. Figure 4 , the device for playing VR video on an extended reality device specifically includes:

[0098] A first display module, configured to open a video playback window in an extended reality space of an extended reality device in response to a user playing a VR video;

[0099] A production module, used for acquiring a VR video image to be played, sampling and rendering the VR video image according to the line of sight of human eyes and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes a 180° or 360° video;

[0100] The second display module is used to display the target image in the video playback window.

[0101] Furthermore, the VR video image is a UV image; the production module is used for:

[0102] Determine the intersection of the line of sight of the human eye and the ball model according to the position of the human eye in the extended reality space, the line of sight of the human eye, the position of the video playback window in the extended reality space, and the position of the ball model in the extended reality space;

[0103] Based on the intersection of the line of sight of the human eye and the ball model, sampling the VR video image to obtain a sampling result;

[0104] Rendering is performed based on the sampling result to obtain a target image.

[0105] Furthermore, the production module is used to:

[0106] Determining a sight beam according to the position of a human eye in the extended reality space and the sight direction of the human eye; the sight beam includes a plurality of sights;

[0107] Based on the position of the video playback window in the extended reality space, determining a target line of sight among the multiple lines of sight; the target line of sight is a line of sight that can pass through the video playback window;

[0108] Determine the intersection point between the target sight line and the ball model;

[0109] Based on the intersection of the target sight line and the ball model, the VR video image is sampled to obtain a sampling result.

[0110] Furthermore, the production module is used to:

[0111] Based on the human eye position in the Cartesian coordinate system, the sight direction of the target sight, and the spherical equation of the ball model, the coordinate value of the intersection point of the target sight and the ball model in the Cartesian coordinate system is obtained;

[0112] Converting the coordinate value of the intersection point of the target sight line and the ball model in the Cartesian coordinate system into the coordinate value of the intersection point of the target sight line and the ball model in the spherical coordinate system;

[0113] Convert the coordinate value of the intersection point of the target sight line and the ball model in the spherical coordinate system to the coordinate value of the intersection point of the target sight line and the ball model in the UV coordinate system;

[0114] Based on the coordinate value of the intersection of the target sight line and the ball model in the UV coordinate system, the VR video image is sampled to obtain a sampling result.

[0115] Furthermore, the device also includes an adjustment module, which is used to:

[0116] In response to a zooming operation on the video playback window, the size of the video playback window and the position and size of the ball model are adjusted so that the VR video image content observed through the video playback window is the same before and after the zooming operation on the video playback window, while the relative position of the human eye and the video playback window remains unchanged.

[0117] Furthermore, the adjustment module is used to:

[0118] In response to a scaling operation on the video playback window, determining a scaling factor of the video playback window;

[0119] Determining a target size of the video playback window and a zoom factor of the ball model based on the zoom factor of the video playback window;

[0120] Determining a target radius and a target center position of the spherical model based on a scaling factor of the spherical model;

[0121] Adjusting the size of the video playback window to a target size;

[0122] The radius of the spherical model is adjusted to the target radius, and the center of the spherical model is adjusted to the target center position.

[0123] Furthermore, the adjustment module is used to:

[0124] Determining a first viewing angle, where the first viewing angle is a maximum viewing angle of a human eye limited by the video playback window before adjusting the size of the video playback window;

[0125] Based on the first viewing angle and the scaling factor of the video playback window, a scaling factor of the ball model is determined.

[0126] Furthermore, the adjustment module is used to:

[0127] Determining a target radius of the spherical model based on an initial radius of the spherical model and a scaling factor of the spherical model;

[0128] The target center position of the spherical model is determined based on the initial radius of the spherical model, the initial center position of the spherical model and the target radius of the spherical model.

[0129] The device for playing VR videos on an extended reality device provided in the embodiment of the present disclosure can execute the steps in the method for playing VR videos on an extended reality device provided in the method embodiment of the present disclosure, and has execution steps and beneficial effects, which will not be repeated here.

[0130] Figure 5 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 5 , which shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing the embodiment of the present disclosure. The electronic device 1000 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0131] like Figure 5 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 to a random access memory (RAM) 1003 to implement a method for playing a VR video on an extended reality device according to an embodiment of the present disclosure. In RAM 1003, various programs and information required for the operation of the electronic device 1000 are also stored. The processing device 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0132] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange information. Although Figure 5 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0133] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the method for playing VR video on an extended reality device as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0134] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include an information signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated information signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0135] In some embodiments, the client and the server may communicate using any known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any known or future developed network.

[0136] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0137] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0138] In response to the user playing the VR video, opening a video playback window in an extended reality space of the extended reality device;

[0139] Acquire a VR video image to be played, and sample and render the VR video image according to the line of sight of human eyes and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes a 180° or 360° video;

[0140] The target image is displayed in the video playback window.

[0141] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0142] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0145] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:

[0148] one or more processors;

[0149] A memory for storing one or more programs;

[0150] When the one or more programs are executed by the one or more processors, the one or more processors implement any method for playing VR videos on an extended reality device as provided in the present disclosure.

[0151] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any method for playing a VR video on an extended reality device as provided in the present disclosure.

[0152] The embodiments of the present disclosure also provide a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the method for playing VR videos on an extended reality device as described above is implemented.

[0153] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0154] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for playing a VR video on an extended reality device, characterized in that: The method comprises: In response to the user playing the VR video, opening a video playback window in an extended reality space of the extended reality device; Acquire a VR video image to be played, and sample and render the VR video image according to the line of sight of human eyes and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes a 180° or 360° video; The target image is displayed in the video playback window.

2. The method according to claim 1, characterized in that The VR video image is a UV image; the VR video image has a ball model corresponding thereto; the VR video image is sampled and rendered according to the line of sight of a human eye and the position of the video playback window in the extended reality space to obtain a target image, and the method includes: Determine the intersection of the line of sight of the human eye and the ball model according to the position of the human eye in the extended reality space, the line of sight of the human eye, the position of the video playback window in the extended reality space, and the position of the ball model in the extended reality space; Based on the intersection of the line of sight of the human eye and the ball model, sampling the VR video image to obtain a sampling result; Rendering is performed based on the sampling result to obtain a target image.

3. The method according to claim 2, characterized in that Determining the intersection of the line of sight of the human eye and the ball model according to the position of the human eye in the extended real space, the line of sight of the human eye, the position of the video playback window in the extended real space, and the position of the ball model in the extended real space includes: Determining a sight beam according to the position of a human eye in the extended reality space and the sight direction of the human eye; the sight beam includes a plurality of sights; Based on the position of the video playback window in the extended reality space, determining a target line of sight among the multiple lines of sight; the target line of sight is a line of sight that can pass through the video playback window; Determine the intersection point between the target sight line and the ball model; The method of sampling the VR video image based on the intersection of the line of sight of the human eye and the ball model to obtain a sampling result includes: Based on the intersection of the target sight line and the ball model, the VR video image is sampled to obtain a sampling result.

4. The method according to claim 3, characterized in that Determining the intersection point between the target sight line and the ball model includes: Based on the human eye position in the Cartesian coordinate system, the sight direction of the target sight, and the spherical equation of the ball model, the coordinate value of the intersection point of the target sight and the ball model in the Cartesian coordinate system is obtained; Converting the coordinate value of the intersection point of the target sight line and the ball model in the Cartesian coordinate system into the coordinate value of the intersection point of the target sight line and the ball model in the spherical coordinate system; Convert the coordinate value of the intersection point of the target sight line and the ball model in the spherical coordinate system to the coordinate value of the intersection point of the target sight line and the ball model in the UV coordinate system; The step of sampling the VR video image based on the intersection of the target sight line and the ball model to obtain a sampling result includes: Based on the coordinate value of the intersection of the target sight line and the ball model in the UV coordinate system, the VR video image is sampled to obtain a sampling result.

5. The method according to claim 2, characterized in that: Also includes: In response to a zooming operation on the video playback window, the size of the video playback window and the position and size of the ball model are adjusted so that the VR video image content observed through the video playback window is the same before and after the zooming operation on the video playback window, while the relative position of the human eye and the video playback window remains unchanged.

6. The method according to claim 5, characterized in that In response to the scaling operation on the video play window, adjusting the size of the video play window and the position and size of the ball model includes: In response to a scaling operation on the video playback window, determining a scaling factor of the video playback window; Determining a target size of the video playback window and a zoom factor of the ball model based on the zoom factor of the video playback window; Determining a target radius and a target center position of the spherical model based on a scaling factor of the spherical model; Adjusting the size of the video playback window to a target size; The radius of the spherical model is adjusted to the target radius, and the center of the spherical model is adjusted to the target center position.

7. The method according to claim 6, characterized in that The step of determining the scaling factor of the ball model based on the scaling factor of the video playback window includes: Determining a first viewing angle, where the first viewing angle is a maximum viewing angle of a human eye limited by the video playback window before adjusting the size of the video playback window; Based on the first viewing angle and the scaling factor of the video playback window, a scaling factor of the ball model is determined.

8. The method according to claim 7, characterized in that The step of determining a target radius and a target center position of the spherical model based on a scaling factor of the spherical model comprises: Determining a target radius of the spherical model based on an initial radius of the spherical model and a scaling factor of the spherical model; The target center position of the spherical model is determined based on the initial radius of the spherical model, the initial center position of the spherical model and the target radius of the spherical model.

9. A device for playing VR videos on an extended reality device, characterized in that: The device comprises: A first display module, configured to open a video playback window in an extended reality space of an extended reality device in response to a user playing a VR video; A production module, used to obtain a VR video image to be played, and sample and render the VR video image according to the line of sight of human eyes and the position of the video playback window in the extended reality space to obtain a target image; the VR video includes a 180° or 360° video; The second display module is used to display the target image in the video playback window.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images