A VR picture capture method and device for cloud gaming

Abstract

The present application provides a VR screen capture method and device for cloud gaming, the method comprising: obtaining the resolution and frame rate parameters sent by the user end, setting the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, respectively renders the left and right eye VR screens of the cloud game through dual-thread rendering, and captures the screen of the virtual VR device of the cloud game based on the time code; the screen is sent to the user end, and the user end combines the screen into a VR screen display according to the time code. The present application simulates the user end through the cloud, and runs the game based on the virtual VR device, so as to realize the interconnection and interoperability of multiple types of devices.

Description

Technical Field

[0001] The present application seeks to protect a cloud rendering technology, and more particularly, relates to a VR picture capture method for cloud gaming. The present application also relates to a VR picture capture device for cloud gaming. Background Art

[0002] VR (virtual reality technology) encompasses computers, electronic information, and simulation technology. Its basic implementation method is to simulate a virtual environment with a computer to give people a sense of environmental immersion. With the continuous development of social productivity and science and technology, the demand for VR technology in all walks of life is growing. VR technology has also made great progress and has gradually become a new field of science and technology.

[0003] However, VR devices in the market are not interconnected due to many reasons. Each product equipment manufacturer only has VR devices based on its own system, which leads to the following problems:

[0004] VR streaming software can only support specific products and devices, and does not support other brands of products and devices. In addition, the streaming software only supports intranet streaming and does not support running on non-VR devices. Summary of the invention

[0005] In order to solve the technical problems mentioned in the above background technology, the present application provides a VR picture capture method applied to cloud games. The present application also relates to a VR picture capture device applied to cloud games.

[0006] The present application provides a VR screen capture method applied to cloud games, including:

[0007] Obtain the resolution and frame rate parameters sent by the user end, and set the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters;

[0008] The virtual VR device starts the cloud game according to the resolution and frame rate parameters, renders the left and right eye VR images of the cloud game respectively through dual-thread rendering, and captures the images of the virtual VR device of the cloud game based on the time code respectively;

[0009] The images are sent to a user terminal, and the user terminal combines the images into a VR image display according to the time code.

[0010] Optionally, the parameters of the virtual VR device are the same as the user-side parameters.

[0011] Optionally, the virtual VR device includes hiding the VR game window.

[0012] The present application also provides a VR screen capture method applied to cloud games, including:

[0013] Setting the resolution and frame rate on the user side, and obtaining a picture rendered based on the resolution and frame rate from the server side, including: the server side obtains the resolution and frame rate parameters sent by the user side, and sets the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, and captures the picture of the virtual VR device of the cloud game;

[0014] The picture decoding mode is selected based on the user terminal type, and the picture is decoded and rendered to the display screen.

[0015] Optionally, the client display screen format includes: VR mode, single-eye screen mode, and binocular screen mode.

[0016] The present application also provides a VR screen capture device for cloud gaming, comprising:

[0017] A setting module, used to obtain the resolution and frame rate parameters sent by the user end, and set the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters;

[0018] A capture module is used for the virtual VR device to start the cloud game according to the resolution and frame rate parameters, respectively render the left and right eye VR images of the cloud game through dual-thread rendering, and respectively capture the images of the virtual VR device of the cloud game based on the time code;

[0019] The sending module is used to send the pictures to the user end, and the user end combines the pictures into a VR picture display according to the time code.

[0020] Optionally, the parameters of the virtual VR device are the same as the user-side parameters.

[0021] Optionally, the virtual VR device includes hiding the VR game window.

[0022] The present application also provides a VR screen capture device for cloud gaming, comprising:

[0023] A parameter module, used to set the resolution and frame rate on the user side, and obtain a rendered image based on the resolution and frame rate from the server side;

[0024] The rendering module selects the picture decoding mode based on the user terminal type, decodes and renders it to the display screen.

[0025] Optionally, the client display screen format includes: VR mode, single-eye screen mode, and binocular screen mode.

[0026] The advantages of this application compared to the prior art are:

[0027] The present application provides a VR screen capture method for cloud gaming, including: obtaining the resolution and frame rate parameters sent by the user end, setting the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, and respectively renders the left and right eye VR screens of the cloud game through dual-thread rendering, and captures the screen of the virtual VR device of the cloud game based on the time code; the screen is sent to the user end, and the user end combines the screen into a VR screen display according to the time code. The present application simulates the user end through the cloud, and runs the game based on the virtual VR device, so as to realize the interconnection and interoperability of multiple types of devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of VR screen capture used in cloud gaming in this application.

[0029] Figure 2 This is the virtual VR device rendering flow chart in this application.

[0030] Figure 3 It is a schematic diagram of a VR screen capture device used for cloud gaming in this application.

[0031] Figure 4 This is a schematic diagram of the server side of the VR screen capture device used for cloud gaming in this application.

[0032] Figure 5 This is a schematic diagram of the user end of the VR screen capture device used for cloud gaming in this application. DETAILED DESCRIPTION

[0033] The following contents are all examples of specific implementation processes provided for describing in detail the technical solutions to be protected by this application. However, this application may also be implemented in other ways different from the descriptions herein. Persons skilled in the art may, under the guidance of the concepts of this application, adopt different technical means to implement this application. Therefore, this application is not limited to the specific embodiments below.

[0034] The present application provides a VR screen capture method for cloud gaming, including: obtaining the resolution and frame rate parameters sent by the user end, setting the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, and respectively renders the left and right eye VR screens of the cloud game through dual-thread rendering, and captures the screen of the virtual VR device of the cloud game based on the time code; the screen is sent to the user end, and the user end combines the screen into a VR screen display according to the time code. The present application simulates the user end through the cloud, and runs the game based on the virtual VR device, so as to realize the interconnection and interoperability of multiple types of devices.

[0035] Figure 1 This is a flow chart of the VR screen capture server used in cloud gaming in this application.

[0036] Please refer to Figure 1 As shown, S101 obtains the resolution and frame rate parameters sent by the user terminal, and sets the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters.

[0037] The acquisition of the resolution and frame rate parameters shown can read the maximum resolution and parameters supported by the user-side VR device, and send the resolution and parameters to the server in the cloud.

[0038] Preferably, the user manually sets the resolution and frame rate at the user end, and sends the resolution and parameters to the server end in the cloud.

[0039] A virtual VR device is created, wherein the virtual VR device is created according to the type of the user terminal device, and the virtual VR device runs in the server in the cloud. Preferably, if the user terminal is not a device, a virtual non-VR device is created according to the type of the user terminal device.

[0040] In the present application, the parameters of the user-end device required for the creation of the virtual VR device include: image decoding parameters, image parameters, etc.

[0041] The cloud server and the client simultaneously create a plurality of the virtual VR devices, which correspond to different user terminals respectively, and the user terminals and the virtual VR devices have a one-to-one correspondence.

[0042] Please refer to Figure 1 As shown, S102 virtual VR device starts the cloud game according to the resolution and frame rate parameters, renders the left and right eye VR images of the cloud game respectively through dual-thread rendering, and captures the images of the virtual VR device of the cloud game based on the time code respectively.

[0043] After the virtual VR device is created, the cloud game is started and run according to the received resolution and frame rate parameters, and the cloud game runs based on the hardware device of the server.

[0044] Before starting, the server detects the hardware resources required for the cloud game to run under the conditions of the resolution and frame rate, and allocates hardware resources to the virtual VR device accordingly.

[0045] After starting and running the cloud game, the virtual VR device calls the rendering device of the server to render the cloud game image. Preferably, the rendering result is no longer displayed in the virtual VR device, but only stored in the cache.

[0046] The rendering result is retrieved from the cache, and is sent to the user end after being encoded.

[0047] Figure 2 This is the virtual VR device rendering flow chart in this application.

[0048] Please refer to Figure 2 As shown, S201 reads the data to be rendered, and sends the data to be rendered to a rendering device for rendering.

[0049] The rendering is performed based on user operations, including lens movement and selection by the user. During the cloud game, the user sends a control instruction to the virtual VR device, and the virtual VR device retrieves the data to be rendered according to the control instruction, and divides the data to be rendered into left and right eye data, and renders them separately.

[0050] The image to be rendered is divided into left and right eye data in the following manner:

[0051] The first step is to retrieve the data of the image to be rendered;

[0052] The second step is to parse the picture data to be rendered to obtain each frame of data;

[0053] The third step is to calculate the visual difference between the left and right eyes based on the data of two adjacent frames;

[0054] Calculating visual disparity involves:

[0055] Identify objects in two adjacent frames of data and select the same center position of the image data to set the coordinate system;

[0056] Based on the coordinate system, calculating the color difference of each coordinate point;

[0057] The coordinate system is used as the center position of each frame, and the image range is segmented according to the different viewing ranges of the left and right eyes. The expression of the segmentation range is as follows:

[0058]

[0059] The X is a distance value based on the X-axis of the coordinate system, the L is the viewing distance, the Lambda is the viewing angle, and the D is the pupil distance.

[0060] Perform color difference adjustment for each coordinate point according to preset color difference adjustment parameters to generate a pixel image to be verified;

[0061] The fourth step is to verify the pixels and generate left and right eye image data to be rendered.

[0062] S202 renders the data to be rendered.

[0063] In the present application, the VR device divides the data to be rendered into left and right eye data and sends them to different rendering devices for rendering. Specifically, the left and right eye VR images of the cloud game are rendered separately through dual-thread rendering.

[0064] When capturing the time code, the time code is determined according to the international standard time. When capturing the time code, the time code is marked for each frame of the left and right eye images respectively.

[0065] Please refer to Figure 1 As shown, S103 sends the images to the user end, and the user end combines the images into a VR image display according to the time code.

[0066] The picture is formed by combining multiple frame images. Specifically, when the VR picture is transmitted to the user terminal, the user terminal performs animation combination according to the time code sequence of the VR picture, and simultaneously aligns the left and right eye images according to the time code to achieve accurate VR picture display.

[0067] Figure 3 This is a user-side flow chart of the VR screen capture method used in cloud gaming in this application.

[0068] Please refer to Figure 3 As shown, S301 sets the resolution and frame rate on the user side, and obtains a picture rendered based on the resolution and frame rate from the server side, including: the server side obtains the resolution and frame rate parameters sent by the user side, and sets the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, and captures the picture of the virtual VR device of the cloud game.

[0069] The server includes VR devices and other display devices. The resolution and frame rate are the resolution and frame rate on the user's display screen.

[0070] Specifically, the display parameters of the user terminal are retrieved, the resolution and frame rate of the user terminal are read, and the resolution and frame rate are sent to the server terminal. Preferably, the user can set the resolution and frame rate on the user terminal, and send the set resolution and frame rate to the server terminal.

[0071] Please refer to Figure 3 As shown, S302 selects the picture decoding method based on the user terminal type, decodes and renders it to the display screen.

[0072] After the server completes rendering of the image, it encodes it and sends it to the client. The client decodes it based on the encoding method and renders the decoded image onto the display screen for display.

[0073] The present application also provides a VR screen capture device for cloud gaming, including: a server and a user.

[0074] Figure 4 This is a schematic diagram of the server side of the VR screen capture device used for cloud gaming in this application.

[0075] Please refer to Figure 4 As shown, the setting module 401 is used to obtain the resolution and frame rate parameters sent by the user terminal, and set the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters.

[0076] The acquisition of the resolution and frame rate parameters shown can read the maximum resolution and parameters supported by the user-side VR device, and send the resolution and parameters to the server in the cloud.

[0077] Preferably, the user manually sets the resolution and frame rate at the user end, and sends the resolution and parameters to the server end in the cloud.

[0078] A virtual VR device is created, wherein the virtual VR device is created according to the type of the user terminal device, and the virtual VR device runs in the server in the cloud. Preferably, if the user terminal is not a VR device, an external keyboard, mouse or other similar tool is required to simulate the control operation of VR.

[0079] In the present application, the parameters of the user-end device required for the creation of the virtual VR device include: image decoding parameters, image parameters, etc.

[0080] The cloud server and the client simultaneously create a plurality of the virtual VR devices, which correspond to different user terminals respectively, and the user terminals and the virtual VR devices have a one-to-one correspondence.

[0081] Please refer to Figure 4 As shown, the capture module 402 is used for the virtual VR device to start the cloud game according to the resolution and frame rate parameters, render the left and right eye VR images of the cloud game respectively through dual-thread rendering, and capture the images of the virtual VR device of the cloud game based on the time code respectively.

[0082] After the virtual VR device is created, the cloud game is started and run according to the received resolution and frame rate parameters, and the cloud game runs based on the hardware device of the server.

[0083] Before starting, the server detects the hardware resources required for the cloud game to run under the conditions of the resolution and frame rate, and allocates hardware resources to the virtual VR device accordingly.

[0084] After starting and running the cloud game, the virtual VR device calls the rendering device of the server to render the cloud game image. Preferably, the rendering result is not displayed in the virtual VR device, but only stored in the cache.

[0085] The rendering result is retrieved from the cache, and is sent to the user end after being encoded.

[0086] Figure 2 This is the virtual VR device rendering flow chart in this application.

[0087] Please refer to Figure 2 As shown, S201 reads the data to be rendered, and sends the data to be rendered to a rendering device for rendering.

[0088] The rendering is performed based on user operations, including lens movement and selection by the user. During the cloud game, the user sends a control instruction to the virtual VR device, and the virtual VR device retrieves the data to be rendered according to the control instruction, and divides the data to be rendered into left and right eye data, and renders them separately.

[0089] The image to be rendered is divided into left and right eye data in the following manner:

[0090] The first step is to retrieve the data of the image to be rendered;

[0091] The second step is to parse the picture data to be rendered to obtain each frame of data;

[0092] The third step is to calculate the visual difference between the left and right eyes based on the data of two adjacent frames;

[0093] Calculating visual disparity involves:

[0094] Identify objects in two adjacent frames of data and select the same center position of the image data to set the coordinate system;

[0095] Based on the coordinate system, calculating the color difference of each coordinate point;

[0096] The coordinate system is used as the center position of each frame, and the image range is segmented according to the different viewing ranges of the left and right eyes. The expression of the segmentation range is as follows:

[0097]

[0098] The X is a distance value based on the X-axis of the coordinate system, the L is the viewing distance, the Lambda is the viewing angle, and the D is the pupil distance.

[0099] Perform color difference adjustment for each coordinate point according to preset color difference adjustment parameters to generate a pixel image to be verified;

[0100] The fourth step is to verify the pixels and generate left and right eye image data to be rendered.

[0101] S202 renders the data to be rendered.

[0102] In the present application, the VR device divides the data to be rendered into left and right eye data and sends them to different rendering devices for rendering. Specifically, the left and right eye VR images of the cloud game are rendered separately through dual-thread rendering.

[0103] When capturing the time code, the time code is determined according to the international standard time. When capturing the time code, the time code is marked for each frame of the left and right eye images respectively.

[0104] Please refer to Figure 4 As shown, the sending module 403 is used to send the picture to the user end, and the user end combines the pictures into a VR picture display according to the time code.

[0105] The picture is formed by combining multiple frame images. Specifically, when the VR picture is transmitted to the user terminal, the user terminal performs animation combination according to the time code sequence of the VR picture, and simultaneously aligns the left and right eye images according to the time code to achieve accurate VR picture display.

[0106] Figure 5 This is a schematic diagram of the user end of the VR screen capture device used for cloud gaming in this application.

[0107] Please refer to Figure 5 As shown, the parameter module 501 is used to set the resolution and frame rate on the user side, and obtain the picture rendered based on the resolution and frame rate from the server side, including: the server side obtains the resolution and frame rate parameters sent by the user side, and sets the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, and captures the picture of the virtual VR device of the cloud game.

[0108] The server includes VR devices and other display devices. The resolution and frame rate are the resolution and frame rate on the user's display screen.

[0109] Specifically, the display parameters of the user terminal are retrieved, the resolution and frame rate of the user terminal are read, and the resolution and frame rate are sent to the server terminal. Preferably, the user can set the resolution and frame rate on the user terminal, and send the set resolution and frame rate to the server terminal.

[0110] Please refer to Figure 5 As shown, the rendering module 502 selects the picture decoding method based on the user terminal type, decodes and renders it to the display screen.

[0111] After the server completes rendering of the image, it encodes it and sends it to the client. The client decodes it based on the encoding method and renders the decoded image onto the display screen for display.

Claims

1. A VR screen capture method for cloud gaming, characterized in that: include: Obtain the resolution and frame rate parameters sent by the user end, and set the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; The virtual VR device starts the cloud game according to the resolution and frame rate parameters, renders the left and right eye VR images of the cloud game respectively through dual-thread rendering, and captures the images of the virtual VR device of the cloud game based on the time code respectively; The images are sent to a user terminal, and the user terminal combines the images into a VR image for display according to the time code; The dual-thread rendering includes: calculating the left-eye and right-eye visual difference based on two adjacent frames of data, specifically: Identify objects in two adjacent frames of data and set a coordinate system at the center of the image data; Based on the coordinate system, calculating the color difference of each coordinate point; According to the left and right eye viewing angle parameters, viewing distance parameters and pupil distance parameters, the image range is segmented using the following formula: ； The X is the distance value on the X axis of the coordinate system, the L is the viewing distance, and the is the viewing angle, and D is the pupil distance; The color difference of each coordinate point is adjusted according to the preset color difference adjustment parameters to generate a pixel image to be verified; the pixels are verified to generate left and right eye rendering picture data.

2. The VR screen capture method for cloud gaming according to claim 1, characterized in that: The parameters of the virtual VR device are the same as those of the user end.

3. The VR screen capture method for cloud gaming according to claim 1, characterized in that: The virtual VR device includes hiding the VR game window.

4. A VR screen capture method for cloud gaming, characterized in that: include: Setting the resolution and frame rate on the user side, and obtaining a picture rendered based on the resolution and frame rate from the server side, including: the server side obtains the resolution and frame rate parameters sent by the user side, and sets the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, and captures the picture of the virtual VR device of the cloud game; Select the image decoding method based on the user terminal type, decode and render to the display screen; Obtain the resolution and frame rate parameters sent by the user end, and set the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; The virtual VR device starts the cloud game according to the resolution and frame rate parameters, renders the left and right eye VR images of the cloud game respectively through dual-thread rendering, and captures the images of the virtual VR device of the cloud game based on the time code respectively; sends the images to the user end, and the user end combines the images into a VR image display according to the time code; The dual-thread rendering includes: calculating the left-eye and right-eye visual difference based on two adjacent frames of data, specifically: Identify objects in two adjacent frames of data and set a coordinate system at the center of the image data; Based on the coordinate system, calculating the color difference of each coordinate point; According to the left and right eye viewing angle parameters, viewing distance parameters and pupil distance parameters, the image range is segmented using the following formula: ； The X is the distance value on the X axis of the coordinate system, the L is the viewing distance, and the is the viewing angle, and D is the pupil distance; The color difference of each coordinate point is adjusted according to the preset color difference adjustment parameters to generate a pixel image to be verified; the pixels are verified to generate left and right eye rendering picture data.

5. The VR screen capture method for cloud gaming according to claim 4, characterized in that: The user terminal display image formats include: VR mode, single-eye image mode, and binocular image mode.

6. A VR picture capture device for cloud gaming, characterized in that: include: A setting module, used to obtain the resolution and frame rate parameters sent by the user end, and set the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; A capture module is used for the virtual VR device to start the cloud game according to the resolution and frame rate parameters, respectively render the left and right eye VR images of the cloud game through dual-thread rendering, and respectively capture the images of the virtual VR device of the cloud game based on the time code; A sending module, used to send the pictures to a user terminal, and the user terminal combines the pictures into a VR picture for display according to the time code; The dual-thread rendering includes: calculating the left-eye and right-eye visual difference based on two adjacent frames of data, specifically: Identify objects in two adjacent frames of data and set a coordinate system at the center of the image data; Based on the coordinate system, calculating the color difference of each coordinate point; According to the left and right eye viewing angle parameters, viewing distance parameters and pupil distance parameters, the image range is segmented using the following formula: ； The X is the distance value on the X axis of the coordinate system, the L is the viewing distance, and the is the viewing angle, and D is the pupil distance; The color difference of each coordinate point is adjusted according to the preset color difference adjustment parameters to generate a pixel image to be verified; the pixels are verified to generate left and right eye rendering picture data.

7. The VR screen capture device for cloud gaming according to claim 6, characterized in that: The parameters of the virtual VR device are the same as those of the user end.

8. The VR screen capture device for cloud gaming according to claim 6, characterized in that: The virtual VR device includes hiding the VR game window.

9. A VR picture capture device applied to cloud games, characterized in that: include: A parameter module, used to set the resolution and frame rate on the user side, and obtain a picture rendered based on the resolution and frame rate from the server side, including: the server side obtains the resolution and frame rate parameters sent by the user side, and sets the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; the virtual VR device starts the cloud game according to the resolution and frame rate parameters, and captures the picture of the virtual VR device of the cloud game; The rendering module selects the picture decoding method based on the user terminal type, decodes and renders it to the display screen; A setting module, used to obtain the resolution and frame rate parameters sent by the user end, and set the resolution and frame rate parameters of the virtual VR device according to the resolution and frame rate parameters; A capture module is used for the virtual VR device to start the cloud game according to the resolution and frame rate parameters, respectively render the left and right eye VR images of the cloud game through dual-thread rendering, and respectively capture the images of the virtual VR device of the cloud game based on the time code; A sending module, used to send the pictures to a user terminal, and the user terminal combines the pictures into a VR picture for display according to the time code; The dual-thread rendering includes: calculating the left-eye and right-eye visual difference based on two adjacent frames of data, specifically: Identify objects in two adjacent frames of data and set a coordinate system at the center of the image data; Based on the coordinate system, calculating the color difference of each coordinate point; According to the left and right eye viewing angle parameters, viewing distance parameters and pupil distance parameters, the image range is segmented using the following formula: ； The X is the distance value on the X axis of the coordinate system, the L is the viewing distance, and the is the viewing angle, and D is the pupil distance; The color difference of each coordinate point is adjusted according to the preset color difference adjustment parameters to generate a pixel image to be verified; the pixels are verified to generate left and right eye rendering picture data.

10. The VR screen capture device for cloud gaming according to claim 9, characterized in that: The user terminal display image formats include: VR mode, single-eye image mode, and binocular image mode.

Images