Data plug flow method and virtual scene display method

Abstract

The invention discloses a data plug flow method and a virtual scene display method. The method comprises the steps that under the condition that a target server establishes connection with a virtual scene display terminal, in response to an application operation request sent by the virtual scene display terminal, a target application and a plug flow client matched with the target application are operated on the target server; under the condition that operation data sent by the virtual scene display terminal are received, the operation data are forwarded to the target application through the flow pushing client side, and the operation data are used for controlling a virtual role in the target application; under the condition that scene data sent by the target application is received, a virtual scene in the target application is rendered according to the scene data, a target picture frame is obtained, and the scene data is used for indicating the virtual scene where the virtual character is located currently; and pushing the coded data of the target picture frame to the virtual scene display terminal through the stream pushing client. The technical problem that an existing data flow pushing method is low in operation efficiency is solved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data streaming method and a virtual scene display method. Background Art

[0002] With the continuous development of XR (Extended Reality) technology, users can experience more and more XR applications through XR devices. Nowadays, users usually experience XR applications through a local LAN via wired USB or wireless WIFI, connecting the XR headset to the local terminal, and then streaming the XR application running on the local terminal to the XR terminal (for example, the XR headset device) through the local terminal.

[0003] Based on the above method, if users in different places want to share XR applications, they need to package and transmit the game before sending the XR application to other users for experience; in addition, the above local streaming method cannot be used if users in different places want to experience XR applications together. In other words, the existing XR data streaming method has the technical problem of low streaming efficiency.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a data streaming method and a virtual scene display method, so as to at least solve the technical problem of low operating efficiency of the existing data streaming method.

[0006] According to one aspect of an embodiment of the present invention, a data streaming method is provided, comprising: when a target server establishes a connection with a virtual scene display terminal, in response to an application running request sent by the virtual scene display terminal, running a target application and a streaming client matching the target application on the target server; when operation data sent by the virtual scene display terminal is received, forwarding the operation data to the target application through the streaming client, wherein the operation data is used to control a virtual character in the target application; when scene data sent by the target application is received, rendering the virtual scene in the target application according to the scene data to obtain a target picture frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located; and pushing the encoded data of the target picture frame to the virtual scene display terminal through the streaming client.

[0007] According to another aspect of an embodiment of the present invention, a method for displaying a virtual scene is provided, comprising: a data receiving client running on a virtual scene display terminal; in a case where the virtual scene display terminal establishes a connection with a target server, receiving, through the data receiving client, encoded data sent by a streaming client running on the target server; displaying a virtual scene in which a virtual character controlled by a current object in a target application is located according to a decoding result of the encoded data; in a case where the virtual scene display terminal receives operation data, forwarding the operation data to the streaming client through the data receiving client, wherein the operation data is used to control the virtual character in the target application.

[0008] According to another aspect of an embodiment of the present invention, a data acquisition device is also provided, including: an operation unit, which is used to run a target application and a streaming client matching the target application on the target server in response to an application operation request sent by the virtual scene display terminal when a connection is established between the target server and the virtual scene display terminal; a forwarding unit, which is used to forward the operation data sent by the virtual scene display terminal to the target application through the streaming client when the operation data is received, wherein the operation data is used to control a virtual character in the target application; a rendering unit, which is used to render the virtual scene in the target application according to the scene data when the scene data is received to obtain a target picture frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located; and a streaming unit, which is used to push the encoded data of the target picture frame to the virtual scene display terminal through the streaming client.

[0009] Optionally, the data streaming device further includes: a first creation unit, for creating a virtual hardware device through the streaming client in response to a virtual device creation request sent by the target application when the target application matches the first streaming mode, wherein the virtual hardware device is used to provide a rendering interface; a second creation unit, for creating a virtual canvas through the streaming client in response to a canvas creation request sent by the target application, wherein the virtual canvas is used to render the virtual scene in the target application based on the scene data through the rendering interface.

[0010] Optionally, the rendering unit is used to: in response to a first rendering instruction sent by the target application, send the canvas identifier of the virtual canvas to the target application through the streaming client, wherein the first rendering instruction is used to obtain the canvas identifier of the virtual canvas in an idle state; receive the scene data sent by the target application, and perform scene rendering on the virtual scene in the target application based on the scene data on the virtual canvas indicated by the canvas identifier; and upon receiving a second rendering instruction sent by the target application, determine the target screen frame according to the virtual canvas when the second rendering instruction indicates that the scene rendering has been completed.

[0011] Optionally, the rendering unit is further used to: when multiple virtual canvases are created through the streaming client and the first canvas among the multiple virtual canvases is used to render the target picture frame, obtain a reference picture frame rendered by the second canvas; perform video encoding according to the reference picture frame, and push the encoded data of the reference picture frame to the virtual scene display terminal through the streaming client.

[0012] Optionally, the rendering unit is used for at least one of the following: in the case of obtaining the perspective operation data sent by the virtual scene display terminal, in response to the perspective data acquisition request sent by the target application, forwarding the perspective data to the target application, wherein the perspective operation data is used to indicate the object perspective that matches the current terminal posture of the virtual scene display terminal; in the case of obtaining the position operation data sent by the virtual scene display terminal, in response to the position data acquisition request sent by the target application, forwarding the position operation data to the target application, wherein the position operation data is used to indicate the object position that matches the current spatial position of the virtual scene display terminal; in the case of obtaining the action operation data sent by the virtual scene display terminal, in response to the action data acquisition request sent by the target application, forwarding the action operation data to the target application, wherein the action operation data is used to indicate the object action detected by the virtual scene display terminal.

[0013] Optionally, the data streaming device further includes: a reference running unit, used to run a scene rendering client on the target server when the target application matches the second streaming mode, wherein the scene rendering client is used to render the virtual scene in the target application according to the scene data; a process adding unit, used to add the streaming process of the streaming client to the running process of the scene rendering client through the scene rendering client, wherein the streaming process is used to obtain the target screen frame rendered by the scene rendering client, and forward the target screen frame to the running process of the streaming client.

[0014] Optionally, the forwarding unit is used to: forward the received operation data to the streaming process through the running process of the streaming client; forward the operation data to the running process of the scene rendering client through the streaming process, wherein the scene rendering client is used to forward the operation data to the target application.

[0015] Optionally, the data streaming device is also used for: when the target server is configured with multiple graphics cards, running the target number of virtual machines on the target server according to the target number of graphics cards configured by the target server, wherein the virtual machines are matched one-to-one with the graphics cards; and running one of the target applications and the streaming client on each of the multiple virtual machines, wherein each of the virtual machines is connected to one of the virtual scene display terminals.

[0016] Optionally, the data streaming device is used to: run a target development application when the target application is in an editing state, wherein the target development application is used to edit the target application; and run the target application in a preview state according to the target development application.

[0017] According to another aspect of an embodiment of the present invention, a display device for a virtual scene is also provided, including: an operating unit, for a data receiving client running on a virtual scene display terminal; a receiving unit, for receiving, through the data receiving client, encoded data sent by a streaming client running on a target server when the virtual scene display terminal establishes a connection with the target server; a display unit, for displaying, according to a decoding result of the encoded data, a virtual scene in which a virtual character controlled by a current object in a target application is located; and a control unit, for forwarding the operation data to the streaming client through the data receiving client when the virtual scene display terminal receives the operation data, wherein the operation data is used to control the virtual character in the target application.

[0018] Optionally, the display device of the above-mentioned virtual scene also includes: a selection unit, which is used to send an establishment request to the server cluster corresponding to the above-mentioned target area in response to a selection operation of the target area from multiple candidate areas; a connection unit, which is used to establish a connection with the above-mentioned streaming client currently running on the target server through the above-mentioned data receiving client when the server cluster corresponding to the above-mentioned target area includes the above-mentioned target server in an idle state.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data streaming method or virtual scene display method when running.

[0020] According to another aspect of the embodiments of the present application, a computer program product is provided, the computer program product including a computer program / instruction, the computer instruction being stored in a computer-readable storage medium. A processor of a computer device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the computer device executes the above data streaming method or virtual scene display method.

[0021] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data streaming method or the virtual scene display method through the computer program.

[0022] In an embodiment of the present invention, when a connection is established between a target server and a virtual scene display terminal, a target application and a streaming client matching the target application are run on the target server in response to an application running request sent by the virtual scene display terminal; when operation data sent by the virtual scene display terminal is received, the operation data is forwarded to the target application through the streaming client, wherein the operation data is used to control a virtual character in the target application; when scene data is received from the target application, the virtual scene in the target application is rendered according to the scene data to obtain a target picture frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located; and the encoded data of the target picture frame is pushed to the virtual scene display terminal through the streaming client, thereby realizing data streaming to the virtual scene display terminal through the streaming client running on the cloud server.

[0023] In the above-mentioned implementation mode of the present application, the game application and the streaming client can be run on the cloud server at the same time, and the encoded data of the virtual scene can be sent to the virtual scene display terminal through the streaming client, and the operation data received from the virtual scene display terminal can be forwarded to the game application, thereby avoiding the packaging and transmission of the VR application, and the VR application can be shared directly with other users in the cloud, and different users can experience the VR application at the same time in different locations, thereby solving the technical problem of low efficiency of the existing data streaming method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 is a schematic diagram of a hardware environment of an optional data streaming method according to an embodiment of the present invention;

[0026] Figure 2 is a flowchart of an optional data streaming method according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of an optional data streaming method according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of another optional data streaming method according to an embodiment of the present invention;

[0029] Figure 5 is a schematic diagram of another optional data streaming method according to an embodiment of the present invention;

[0030] Figure 6 is a schematic diagram of another optional data streaming method according to an embodiment of the present invention;

[0031] Figure 7 is a schematic diagram of another optional data streaming method according to an embodiment of the present invention;

[0032] Figure 8 is a schematic diagram of another optional data streaming method according to an embodiment of the present invention;

[0033] Fig. 9 is a schematic diagram of another optional data streaming method according to an embodiment of the present invention;

[0034] Fig.10 is a schematic diagram of another optional data streaming method according to an embodiment of the present invention;

[0035] Fig.11 is a schematic diagram of an interface of an optional virtual scene display method according to an embodiment of the present invention;

[0036] Fig.12 is a schematic structural diagram of an optional data streaming device according to an embodiment of the present invention;

[0037] Fig.13 is a schematic structural diagram of an optional electronic device according to an embodiment of the present invention;

[0038] Fig.14is a schematic structural diagram of an optional virtual scene display device according to an embodiment of the present invention;

[0039] Fig.15 is a schematic structural diagram of another optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] The following is an explanation of the technical terms used in this application:

[0043] OpenXR: Image rendering and device access SDK for gaming and VR.

[0044] SteamVR: also known as OpenVR, is the predecessor of the OpenXR standard, formulated and developed by the Steam gaming platform;

[0045] d3d11 and d3d12: Windows graphics rendering toolkit, full name: DirectX or D3D. Currently, most PC games use d3d11, and a few new games use d3d12;

[0046] Game editors: tools used to make games, such as Unity and Unreal Engine (UE4);

[0047] Head-mounted display: refers to VR glasses, sometimes also called helmet or HMD;

[0048] XR: Extended Reality, refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. It is also a general term for multiple technologies such as AR, VR, and MR. By integrating multiple technologies, it brings the experiencer an "immersive feeling" of seamless transition between the virtual world and the real world;

[0049] AR: Augmented Reality, which is a process of superimposing virtual information content on the real world through simulation processing, and can be perceived by human senses in the process. After the real environment and virtual objects overlap, they can exist simultaneously in the same picture and space;

[0050] VR: Virtual Reality, refers to the use of computer technology, and the integration of the latest developments in 3D graphics technology, multimedia technology, simulation technology, display technology, servo technology and other high-tech technologies to create a virtual world with realistic 3D vision, touch, smell and other sensory experiences with the help of computers and other equipment;

[0051] MR: Mixed Reality is a further development of virtual reality technology. This technology introduces real scene information into the virtual environment, builds an interactive feedback information loop between the virtual world, the real world and the user, so as to enhance the realism of the user experience.

[0052] Tracking: VR term, mainly composed of the posture information of the head display HMD and the posture information of the two controllers.

[0053] Cloud computing belongs to the category of distributed computing, which is a form of distributed computing oriented towards providing external services. Cloud computing refers to the delivery and use model of information technology (IT) infrastructure, which means obtaining required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use model of services, which means obtaining required services through the network in an on-demand and easily scalable manner. This service can be IT and software, Internet-related, or other services. Cloud computing is the product of the integration of the development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0054] According to one aspect of an embodiment of the present invention, a data streaming method is provided. As an optional implementation, the data streaming method can be applied to, but is not limited to, Figure 1 The data streaming system in the hardware environment shown in FIG. 1 may include but is not limited to a terminal device 102, a network 104, a server 106, a database 108, and a terminal device 110. The terminal device 102 runs a data receiving client (such as Figure 1 As shown, the target client is a data receiving client of a cloud game application as an example). The terminal device 102 includes a human-computer interaction screen, a processor and a memory. The human-computer interaction screen is used to display a virtual game scene (such as Figure 1 The terminal device 102 is used to store relevant attribute data, such as object attribute information of the controlled virtual object and attribute information of the held virtual props, etc. The attribute information here may include but is not limited to information for identifying its identity and current location. Among them, a data receiving client of a cloud gaming application is running in the terminal device 102, and virtual objects can be controlled to complete virtual tasks in the cloud gaming application. The virtual objects here can be controlled to perform certain interactive events in the game, such as attack events, defense events, skill casting events, etc. It should be noted that, in this embodiment, the above-mentioned terminal device 102 can be used as a display terminal, that is, the above-mentioned cloud game application is actually running in the server 106, and the above-mentioned terminal device 102 is used to display the running results of the above-mentioned game application in the display interface; in addition, the above-mentioned terminal device 110 can be another terminal device running the above-mentioned data receiving client, and the above-mentioned terminal device 110 can also be connected to the server 106 and receive the push stream of the server 106.

[0055] The server 106 may be a server for running the cloud gaming application and the streaming client. For example, the server 106 may be responsible for rendering the game screen and sending the rendered game screen to the client running on the terminal device 102 through the streaming client for display.

[0056] In addition, the server 106 includes a processing engine, which is used to perform storage or reading operations on the database 108. Specifically, the processing engine reads virtual scene information of each virtual object and operation information performed by each virtual object from the database 108.

[0057] Assumptions Figure 1The terminal device 102 is used to display the virtual game scene, and may be a virtual scene display terminal. The specific process of this embodiment is as follows:

[0058] As in step S102, the terminal device 102 sends a connection request to the server 106 through the network 104; then the server 106 executes S104 and S106 to establish a connection with the virtual scene display terminal; when the target server is connected to the virtual scene display terminal, in response to the application running request sent by the virtual scene display terminal, the target application and the streaming client matching the target application are run on the target server;

[0059] Then, as in steps S108 and S110, the terminal device 102 generates operation data according to the user operation; sends the operation data to the server 106 through the network 104; then the server 106 executes S112 to S114 and forwards the operation data to the target application through the streaming client upon receiving the operation data sent by the virtual scene display terminal, wherein the operation data is used to control the virtual character in the target application; upon receiving the scene data sent by the target application, the virtual scene in the target application is rendered according to the scene data to obtain a target picture frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located; and pushes the encoded data to the terminal device 102 through the streaming client network 104, wherein the encoded data is obtained by performing an encoding operation according to the target picture frame.

[0060] Finally, the terminal device 102 performs S116 to decode the encoded data to display the virtual scene.

[0061] Optionally, in this embodiment, the specific type of the above-mentioned terminal device may include but is not limited to at least one of the following: XR device, VR device, AR (augment reality) device, MR (mixed reality) device. The above-mentioned network may include but is not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The above-mentioned server can be a single server, or a server cluster composed of multiple servers, or a cloud server. The above is only an example, and this embodiment does not make any limitation to this.

[0062] Optionally, in this embodiment, the data push method can be applied to a cloud VR game application scenario, a cloud XR game scenario, etc., but not limited to the cloud VR game. The cloud VR game can be a game terminal application (Application, referred to as APP) that completes a predetermined confrontation game task in a virtual scene, such as a virtual confrontation game application in a multiplayer online tactical competitive game (Multiplayer Online Battle Arena, referred to as MOBA) application; the confrontation game task can be, but not limited to, a game task completed by the current player through confrontation interaction with the virtual objects in the virtual scene through human-computer interaction data push; the data push method can also be applied to a massively multiplayer online role-playing game (Multiplayer Online Role-Playing Game, referred to as MMORPG) terminal application, in which the current player can complete the social game task in the game from the first perspective of the virtual object through role-playing, for example, completing the game task together with other virtual objects. The social game task here can be, but not limited to, running in an application (such as a non-independent game APP) in the form of a plug-in or applet, or running in an application (such as an independent game APP) in a game engine. The types of the above-mentioned game applications may include but are not limited to at least one of the following: two-dimensional (2D) game applications, three-dimensional (3D) game applications, virtual reality (VR) game applications, augmented reality (AR) game applications, and mixed reality (MR) game applications. The above is only an example, and this embodiment does not impose any limitation on this.

[0063] In an embodiment of the present invention, when a connection is established between a target server and a virtual scene display terminal, a target application and a streaming client matching the target application are run on the target server in response to an application running request sent by the virtual scene display terminal; when operation data sent by the virtual scene display terminal is received, the operation data is forwarded to the target application through the streaming client, wherein the operation data is used to control a virtual character in the target application; when scene data is received from the target application, the virtual scene in the target application is rendered according to the scene data to obtain a target picture frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located; and the encoded data of the target picture frame is pushed to the virtual scene display terminal through the streaming client, thereby realizing data streaming to the virtual scene display terminal through the streaming client running on the cloud server.

[0064] In the above-mentioned implementation mode of the present application, the game application and the streaming client can be run on the cloud server at the same time, and the encoded data of the virtual scene can be sent to the virtual scene display terminal through the streaming client, and the operation data received from the virtual scene display terminal can be forwarded to the game application, thereby avoiding the packaging and transmission of the VR application, and the VR application can be shared directly with other users in the cloud, and different users can experience the VR application at the same time in different locations, thereby solving the technical problem of low efficiency of the existing data streaming method.

[0065] As an optional implementation, Figure 2 As shown, the above data streaming method can be applied to a cloud server, including the following steps:

[0066] S202, when the target server is connected to the virtual scene display terminal, in response to an application running request sent by the virtual scene display terminal, running the target application and a streaming client matching the target application on the target server;

[0067] S204, when the operation data sent by the virtual scene display terminal is received, the operation data is forwarded to the target application through the streaming client, wherein the operation data is used to control the virtual character in the target application;

[0068] S206, when receiving the scene data sent by the target application, rendering the virtual scene in the target application according to the scene data to obtain a target screen frame, wherein the scene data is used to indicate the virtual scene where the virtual character is currently located;

[0069] S208: Push the encoded data of the target picture frame to the virtual scene display terminal through the streaming client.

[0070] It should be noted that the virtual scene display terminal in the above step S202 may include but is not limited to a VR device, an AR (augment reality) device, an MR (mixed reality) device, an XR device, etc.; the above target server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. In this embodiment, the specific types of the above virtual scene display terminal and the above target server are not limited.

[0071] The above-mentioned target application may include, but is not limited to, a VR cloud game application, an AR cloud navigation application, a VR cloud live broadcast application, etc. In the case where the above-mentioned target application is a VR cloud game application, the user object can display the VR game scene through the display device in the VR device, and control the virtual objects in the game through a signal input device such as a handle; in the case where the above-mentioned target application is an AR cloud navigation application, the user object can display the AR navigation scene through the display device in the AR device (for example, superimposing a virtual navigation path and navigation prompt information on the real scene through AR glasses), and control the virtual objects in the AR navigation scene through the signal input device, such as controlling the movement of the virtual objects in the above-mentioned AR navigation scene; in the case where the above-mentioned target application is a VR cloud live broadcast application, the user object can display the VR live broadcast scene through the display device in the VR device, and interact with the host object and the audience object in the virtual live broadcast scene through the signal input device, such as giving virtual gifts, face-to-face communication, etc. This embodiment does not limit the specific application type of the above-mentioned target application.

[0072] In the above step S204, when the above virtual scene display terminal is specifically a VR device, the operation data sent by the above virtual scene display terminal can be specifically the posture information of the head display HMD and the posture information of the handle, and then the virtual objects in the virtual scene are controlled through the above posture information, such as controlling the virtual objects to move, interact with other virtual objects, and adjust the viewing angle of the virtual objects to display different virtual scene images.

[0073] It should be noted that if Figure 3 As shown, in this embodiment, the streaming client includes at least two modules to implement streaming of different types of VR game applications. Figure 3 As shown, the existing development tools of VR game applications may include but are not limited to Unreal Engine, Unity and native APP development. The push streaming modes of the game applications developed and generated by the above three methods may include two, the first of which is to push streaming through the OpenXR protocol, and the other is to push streaming through SteamVR on the Steam platform. In this embodiment, the above-mentioned push streaming client is compatible with both OpenXR and SteamVR push streaming methods. Specifically, it includes two modules, of which the OpenVR_client module is responsible for implementing the rendering and image acquisition of SteamVR; the OpenXR_client module can render and acquire images directly from the game application without the SteamVR platform.

[0074] It can be understood that, when the streaming method of the above-mentioned VR game application is to stream through the OpenXR protocol, the virtual scene can be rendered based on the game scene data through the OpenXR_client in the above-mentioned streaming client, and the encoded data of the rendered target picture frame can be sent to the above-mentioned virtual scene display terminal through the above-mentioned streaming client; and when the streaming method of the above-mentioned VR game application is to stream through SteamVR, the above-mentioned target picture frame can be rendered through the client corresponding to the above-mentioned SteamVR to obtain the above-mentioned target picture frame.

[0075] Through the above-mentioned implementation mode of the present application, the game application and the streaming client can be run on the cloud server at the same time, and the encoded data of the virtual scene can be sent to the virtual scene display terminal through the streaming client, and the operation data received from the virtual scene display terminal can be forwarded to the game application, thereby avoiding the packaging and transmission of the VR application, and the VR application can be shared directly with other users in the cloud, and different users can experience the VR application at the same time in different locations, thereby solving the technical problem of low efficiency of the existing data streaming method.

[0076] In an optional implementation, when the scene data sent by the target application is received, rendering the virtual scene in the target application according to the scene data to obtain the target screen frame further includes:

[0077] S1, when the target application matches the first streaming mode, in response to a virtual device creation request sent by the target application, creating a virtual hardware device through a streaming client, wherein the virtual hardware device is used to provide a rendering interface;

[0078] S2, in response to a canvas creation request sent by the target application, creating a virtual canvas through the streaming client, wherein the virtual canvas is used to render a virtual scene in the target application based on scene data through a rendering interface.

[0079] It should be noted that the first streaming mode can be specifically the streaming mode for streaming through the OpenXR protocol. In the above mode, the rendering of the virtual scene can be realized through the OpenXR_client in the above streaming client. Since it is directly connected to the game and is responsible for completing the rendering of the game, it is necessary to respond to the creation request of the game application to make relevant preparations for the screen rendering.

[0080] Specifically, Figure 4As shown, in the case where the target application is specifically a VR game application, first, the game application can request OpenXR_client to create a D3D device (i.e., the above-mentioned virtual hardware device) for it through the first interface provided by the OpenXR protocol (specifically, it can be the xrGetD3D11GraphicsRequirementsKHR interface). It can be understood that by creating the above-mentioned D3D device, the interface set provided by the graphics card can be used to render the virtual scene; after OpenXR_client completes the creation of the D3D device, the game application notifies OpenXR_client through the second interface (specifically, it can be the xrCreateSwapchain interface) to create a texture for it for rendering the left and right eye textures (i.e., the above-mentioned virtual canvas) respectively. Finally, when the game renders each frame of the image based on the created texture, it can notify OpenXR_client through the third interface (specifically, it can be the xrEndFrame interface). At this time, a texture can be copied from the texture of the game and passed to the external process for use. For example, the rendered texture (i.e., the above-mentioned target screen frame) is video encoded and sent to the VR device.

[0081] Through the above-mentioned implementation mode of the present application, when the target application matches the first streaming mode, a virtual hardware device is created through the streaming client in response to a virtual device creation request sent by the target application; a virtual canvas is created through the streaming client in response to a canvas creation request sent by the target application, thereby completing the preparation work for data streaming based on the OpenXR protocol through the above-mentioned streaming client, so as to improve the efficiency of subsequent texture rendering and data encoding streaming.

[0082] In an optional implementation, when the scene data sent by the target application is received, rendering the virtual scene in the target application according to the scene data to obtain the target screen frame includes:

[0083] S1, in response to a first rendering instruction sent by a target application, sending a canvas identifier of a virtual canvas to the target application through a streaming client, wherein the first rendering instruction is used to obtain a canvas identifier of a virtual canvas in an idle state;

[0084] S2, receiving scene data sent by the target application, and performing scene rendering on a virtual scene in the target application based on the scene data on a virtual canvas indicated by the canvas identifier;

[0085] S3: upon receiving a second rendering instruction sent by the target application, and when the second rendering instruction indicates that scene rendering is completed, determining a target picture frame according to the virtual canvas.

[0086] In this embodiment, when the target application is an application that performs streaming based on the OpenXR protocol, the target application can render the target screen frame through multiple instructions provided by the protocol.

[0087] Specifically, during the scene rendering process through the above-mentioned streaming client and the above-mentioned game application, the game application can obtain the renderable texture serial number from OpenXR_client through the xrAcquireSwapchainImage interface (that is, the above-mentioned first rendering instruction); after the rendering is completed, the above-mentioned game application can call xrEndFrame to notify OpenXR_client that the texture has been rendered, and record the time when the rendering is completed.

[0088] Through the above-mentioned implementation mode of the present application, in response to the first rendering instruction sent by the target application, the canvas identifier of the virtual canvas is sent to the target application through the streaming client; the scene data sent by the target application is received, and the virtual scene in the target application is rendered based on the scene data on the virtual canvas indicated by the canvas identifier; when the second rendering instruction sent by the target application is received, when the second rendering instruction indicates that the scene rendering is completed, the target screen frame is determined according to the virtual canvas, so that the above-mentioned streaming client responds to the instruction of the game application to directly render the scene on the texture according to the scene data of the game application, thereby improving the scene rendering efficiency.

[0089] In an optional implementation, when the scene data sent by the target application is received, the virtual scene in the target application is rendered according to the scene data by the streaming client to obtain the target screen frame, further comprising:

[0090] S1, when multiple virtual canvases are created by a streaming client, and a first canvas among the multiple virtual canvases is used to render a target picture frame, obtaining a reference picture frame rendered by a second canvas;

[0091] S2, video encoding is performed according to the reference picture frame, and the encoded data of the reference picture frame is pushed to the virtual scene display terminal through the streaming client.

[0092] It can be understood that in this embodiment, when the virtual canvas is specifically a VR texture, multiple textures can be created simultaneously in the creation phase to achieve asynchronous copying and transmission of picture frames through rendering of multiple textures.

[0093] like Figure 5As shown, in the case where the created textures include texture 501, texture 502, and texture 503, after the game application obtains the texture identifiers of the above-mentioned multiple textures, it is assumed that the game application first completes the rendering of the first screen frame through the above-mentioned texture 501, and then the screen frame on the above-mentioned texture 501 can be copied and transmitted through the above-mentioned streaming client; at the same time, the game application can start rendering the second screen frame by obtaining the identifier of the above-mentioned texture 502, thereby realizing asynchronous texture copying and texture rendering, and improving texture rendering efficiency.

[0094] In an optional implementation, when the operation data sent by the virtual scene display terminal is received, forwarding the operation data to the target application through the streaming client includes at least one of the following:

[0095] Method 1: when the perspective operation data sent by the virtual scene display terminal is obtained, in response to the perspective data acquisition request sent by the target application, the perspective data is forwarded to the target application, wherein the perspective operation data is used to indicate the object perspective matching the current terminal posture of the virtual scene display terminal;

[0096] Method 2: when the position operation data sent by the virtual scene display terminal is obtained, in response to the position data acquisition request sent by the target application, the position operation data is forwarded to the target application, wherein the position operation data is used to indicate the object position matching the current spatial position of the virtual scene display terminal;

[0097] Method three: when the action operation data sent by the virtual scene display terminal is obtained, the action operation data is forwarded to the target application in response to the action data acquisition request sent by the target application, wherein the action operation data is used to indicate the object action detected by the virtual scene display terminal.

[0098] It is understandable that in this embodiment, the game application can obtain relevant operation data from the above-mentioned streaming client through multiple interfaces. For example, the game application can obtain the perspective operation data through the xrLocateViews interface provided by the above-mentioned OpenXR protocol, which is used to simulate the up, down, left, and right changes of the human head to adjust the scene perspective of the virtual scene. It can also obtain position operation data through the xrLocateSpace interface provided by the OpenXR protocol to simulate the position change of the virtual object. In addition, action operation data can also be obtained from the streaming client through other interface methods, such as the handle operation data received by the VR device, to control the virtual character in the virtual scene to perform various virtual operations.

[0099] In addition, before notifying OpenXR_client to perform drawing and rendering operations, the game application can also call the xrWaitFrame interface and the xrBenginFrame interface to instruct OpenXR_client to record the rendering time and frame number of each frame; after the game application completes the rendering based on the above texture, it will call the xrEndFrame interface to notify OpenXR_client that the texture has been rendered and record the time when the rendering is completed. The rendering frame rate can be controlled through the above three OpenXR interfaces. For example, developers can choose to configure a rendering frame rate of 60FPS or 90FPS in the above streaming client, and then through the combined control of the above three interfaces, they can control the rendering cycle of the game according to the specified frame rate.

[0100] In addition, in the above-mentioned game application, the xrWaitSwapchainImage interface can be used to notify OpenXR_client to control the texture selected by the game application to enter the waiting drawing state. It is understandable that OpenGL and Vulkan can have different implementation modes to put the texture into the waiting drawing mode; after all drawing is completed, the game application can call the xrReleaseSwapchain interface to notify OpenXR_client to release the texture, and finally call the xrEndFrame interface to indicate that the current frame of the image has been drawn. After the game application calls xrEndFrame, the above-mentioned streaming client can add the texture to the cache queue, and then perform video encoding and send it to the virtual scene display terminal (for example, a head-mounted display device) for decoding and playback.

[0101] like Figure 6 As shown, in a specific implementation, the streaming client can render on two graphics devices, d3d11 and d3d12, and the game developer can switch between different graphics devices according to their needs.

[0102] In an optional implementation, when the scene data sent by the target application is received, the virtual scene in the target application is rendered by the streaming client according to the scene data to obtain the target screen frame, and the method further includes:

[0103] S1, when the target application matches the second streaming mode, running a scene rendering client on the target server, wherein the scene rendering client is used to render a virtual scene in the target application according to the scene data;

[0104] S2, adding the streaming process of the streaming client to the running process of the scene rendering client through the scene rendering client, wherein the streaming process is used to obtain the target picture frame rendered by the scene rendering client, and forward the target picture frame to the running process of the streaming client.

[0105] In an optional implementation, when the operation data sent by the virtual scene display terminal is received, forwarding the operation data to the target application through the streaming client includes:

[0106] S1, forwarding the received operation data to the streaming process through the running process of the streaming client;

[0107] S2, forwarding the operation data to the running process of the scene rendering client through the streaming process, wherein the scene rendering client is used to forward the operation data to the target application.

[0108] It is understandable that in this embodiment, the above-mentioned target application is a game application for streaming based on SteamVR on the Steam platform. In this embodiment, the SteamVR client and the above-mentioned streaming client can be run simultaneously on the target server, and the external call interface provided by the SteamVR client can be used to synchronize the processes of the two. It is understandable that SteamVR internally implements two rendering modes of OpenVR and OpenXR, but ultimately provides external calls in the form of OpenVR, so that it can be compatible with those old games and streaming tools.

[0109] like Figure 7 As shown in the figure, taking the VR game streaming method based on SteamVR client as an example, UNITY, Unreal Engine, and native applications are all games or game editors. They can load corresponding plug-ins through the OpenVR or OpenXR protocol, that is, Figure 7 The OpenXR plug-in (OpenXR_plugin) and OpenVR plug-in (OpenVR_plugin) in the SteamVR process can communicate with the SteamVR process. Once the communication is established, the rendering, frame rate and Tracking of the game are all controlled by SteamVR. OpenVR_client.dll is the streaming process loaded by the streaming client in the SteamVR process in this embodiment, which is used to obtain the left and right eye textures rendered by the game application from SteamVR and forward them to the main process of the streaming client, and then perform video encoding and streaming to the head-mounted display device through the main process of the streaming client. In addition, when the main process of the streaming client obtains the Tracking information of the head display, it communicates with OpenVR_client.dll and writes the Tracking to the game application through SteamVR to achieve the movement of the game perspective and the control of the handle.

[0110] like Figure 8As shown, another streaming architecture diagram involved in the present application is shown. The streaming process of the streaming client can be used to receive the Tracking information returned by the head-mounted display device and cache it in the local queue. Then, the SteamVR process obtains information from the Tracking information cached in the local queue in a certain order through the OpenVR_client.dll process added to its process, and renders the game screen based on the acquired Tracking information. Then the virtual scene rendered in the left and right eye textures is sent to the streaming process of the streaming client through OpenVR_client.dll. It can be understood that after obtaining the texture data, the acquired picture frames can also be cached through the cache queue in the streaming process, and encoded and pushed to the head-mounted display device in a certain order.

[0111] In an optional implementation, when the target server is connected to the virtual scene display terminal, before running the target application and the streaming client matching the target application on the target server in response to the application running request sent by the virtual scene display terminal, the method further includes:

[0112] S1, when the target server is configured with multiple graphics cards, running a target number of virtual machines on the target server according to the target number of graphics cards configured on the target server, wherein the virtual machines are matched one by one with the graphics cards;

[0113] S2, running a target application and a streaming client on multiple virtual machines respectively, wherein each virtual machine establishes a connection with a virtual scene display terminal respectively.

[0114] Understandably, if Fig. 9 As shown, in the cloud server, each server can contain 2 or more independent graphics cards. Since Windows games are screen-exclusive, and SteamVR and Unity game editors use the main graphics card as the rendering device, they will not actively use the secondary graphics card for scene rendering. In the case where the server includes two independent graphics cards, in order to make full use of the independent graphics cards on the cloud server, you can create two Windows virtual machines on the server to use these two graphics cards for rendering respectively, and realize multi-game opening by virtualizing two operating systems corresponding to user 1 and user 2 respectively. In this way, the rendering resources of the cloud server can be fully utilized.

[0115] In an optional implementation, when the target server is connected to the virtual scene display terminal, in response to the application execution request sent by the virtual scene display terminal, executing the target application and the streaming client matching the target application on the target server includes:

[0116] S1, when the target application is in an editing state, running a target development application, wherein the target development application is used to edit the target application;

[0117] S2, running the target application in the preview state according to the target development application.

[0118] It is understandable that, in this embodiment, the above streaming method can be applied to game preview during game development. During VR game development, developers use game editors to create VR games, and can experience the effects and interactions of each game map in real time through the "VR Preview" button of the game editor.

[0119] In the development process of some cross-regional cooperation, if developers in other regions want to experience the content of a certain game level, they must package the game into an executable file (exe or apk) and send it to other people for installation and experience. Since game assets are generally large (usually dozens or hundreds of G), the packaging time is also very long, so the scene of temporarily giving someone an experience has always been time-consuming. Therefore, through the above-mentioned implementation method of the present application, VR streaming tools, game editors and games are deployed in the cloud. Users who want to experience only need to install a VR content player on the head display side and connect to the remote server to experience the content. There is no need to package the game at all, and there is no need to prepare an additional Windows PC for data streaming. Thereby improving the data streaming efficiency of VR game applications.

[0120] The following combination Fig.10 A specific example of the above-mentioned implementation method of the present application is described.

[0121] In this embodiment, a streaming client running on a server is provided, which can support both OpenXR streaming and SteamVR streaming; Fig.10 As shown, developers can select the "streaming platform selection" option as needed;

[0122] When the cloud server chooses to push the stream with SteamVR, the streaming client will pull up the SteamVR process and register the OpenVR_client of the streaming tool to the SteamVR environment. OpenVR_client can obtain the left and right eye textures of the game from SteamVR, and send the textures to the streaming client for video encoding, and finally send the data to the head display device for decoding and display.

[0123] When the cloud server chooses to push the stream with OpenXR, the cloud server can open the game editor (such as UE) and connect directly to the game editor through the push streaming client. The game application can communicate with the push streaming client through the control interface provided by OpenXR, and then complete the rendering of the game screen, the time prediction of the next frame, and the writing of input device events to the direct editor through the push streaming client. Finally, the current level screen is pushed to the headset through the push streaming client.

[0124] In an optional implementation, the present application provides a method for displaying a virtual scene on a virtual scene display device, comprising the following steps:

[0125] S1, a data receiving client running on a virtual scene display terminal;

[0126] S2, when the virtual scene display terminal is connected to the target server, receiving the encoded data sent by the streaming client running on the target server through the data receiving client;

[0127] S3, displaying the virtual scene where the virtual character controlled by the current object in the target application is located according to the decoding result of the above-mentioned encoded data;

[0128] S4, when the virtual scene display terminal receives the operation data, the operation data is forwarded to the streaming client through the data receiving client, wherein the operation data is used to control the virtual character in the target application.

[0129] Optionally, in the case where the virtual scene display terminal establishes a connection with the target server, before the data receiving client receives the encoded data sent by the streaming client running on the target server, the method includes:

[0130] S1, in response to a selection operation of a target area from a plurality of candidate areas, sending a creation request to a server cluster corresponding to the target area;

[0131] S2: When the server cluster corresponding to the target area includes the target server in an idle state, a connection is established between the data receiving client and the streaming client currently running on the target server.

[0132] In this embodiment, after the user installs the player APK on the head display side, he can access the cloud server to experience VR content by selecting a seat. Fig.11In the interface shown, users can select a server with a shorter physical distance for streaming according to their region. Each cloud server in each region has several seats for user experience, such as Region A, Region B, and Region C. When a user enters one of the seats, the seat becomes red and busy, and the remaining users can no longer enter this seat to control the game. After the user accesses the seat, he can see the VR game screen and use his own controller to control the game.

[0133] It should be noted that the methods for acquiring various types of data in the above-mentioned implementation modes of the present application comply with the provisions of relevant normative documents, and before acquiring data related to a user object, it is necessary to obtain permission and authorization from the corresponding user object.

[0134] Through the above implementation of the present application, a streaming client is provided, which can be used as a collaborative platform for game developers. During the game development process, game levels can be shared with other people for direct experience, without the need to package the game and send it to the user for experience, and users do not need to configure the PC and LAN environment locally. It can save manpower and costs and improve streaming efficiency. The above streaming client supports both SteamVR streaming and OpenXR rendering and streaming; the above client is adapted to both d3d11 and d3d12 Windows graphics interfaces in terms of OpenXR rendering. D3d11 is used for rendering of older game editors (such as UE4.27), and d3d12 is used for newer editors (such as UE5.1). This can maximize support for different game projects.

[0135] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0136] According to another aspect of the embodiments of the present invention, a data streaming device for implementing the above-mentioned data streaming method is also provided. Fig.12 As shown, the device comprises:

[0137] The running unit 1202 is used to run the target application and the streaming client matching the target application on the target server in response to the application running request sent by the virtual scene display terminal when the target server is connected to the virtual scene display terminal;

[0138] The forwarding unit 1204 is used for forwarding the operation data sent by the virtual scene display terminal to the target application through the streaming client when the operation data is received, wherein the operation data is used to control the virtual character in the target application;

[0139] The rendering unit 1206 is used to render the virtual scene in the target application according to the scene data to obtain a target screen frame when the scene data sent by the target application is received, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located;

[0140] The streaming unit 1208 is used to push the encoded data of the target picture frame to the virtual scene display terminal through the streaming client.

[0141] Optionally, the data streaming device further includes: a first creation unit, for creating a virtual hardware device through the streaming client in response to a virtual device creation request sent by the target application when the target application matches the first streaming mode, wherein the virtual hardware device is used to provide a rendering interface; a second creation unit, for creating a virtual canvas through the streaming client in response to a canvas creation request sent by the target application, wherein the virtual canvas is used to render the virtual scene in the target application based on the scene data through the rendering interface.

[0142] Optionally, the rendering unit 1206 is configured to: in response to a first rendering instruction sent by the target application, send a canvas identifier of the virtual canvas to the target application through the streaming client, wherein the first rendering instruction is used to obtain the canvas identifier of the virtual canvas in an idle state; receive the scene data sent by the target application, and perform scene rendering on the virtual scene in the target application based on the scene data on the virtual canvas indicated by the canvas identifier; and upon receiving a second rendering instruction sent by the target application, determine the target screen frame according to the virtual canvas when the second rendering instruction indicates that the scene rendering has been completed.

[0143] Optionally, the rendering unit 1206 is further used to: when a plurality of the virtual canvases are created through the streaming client, and a first canvas among the plurality of virtual canvases is used to render the target picture frame, obtain a reference picture frame rendered by a second canvas; perform video encoding according to the reference picture frame, and push the encoded data of the reference picture frame to the virtual scene display terminal through the streaming client.

[0144] Optionally, the rendering unit 1206 is used for at least one of the following: in the case of obtaining the perspective operation data sent by the virtual scene display terminal, in response to the perspective data acquisition request sent by the target application, forwarding the perspective data to the target application, wherein the perspective operation data is used to indicate the object perspective that matches the current terminal posture of the virtual scene display terminal; in the case of obtaining the position operation data sent by the virtual scene display terminal, in response to the position data acquisition request sent by the target application, forwarding the position operation data to the target application, wherein the position operation data is used to indicate the object position that matches the current spatial position of the virtual scene display terminal; in the case of obtaining the action operation data sent by the virtual scene display terminal, in response to the action data acquisition request sent by the target application, forwarding the action operation data to the target application, wherein the action operation data is used to indicate the object action detected by the virtual scene display terminal.

[0145] Optionally, the data streaming device further includes: a reference running unit, used to run a scene rendering client on the target server when the target application matches the second streaming mode, wherein the scene rendering client is used to render the virtual scene in the target application according to the scene data; a process adding unit, used to add the streaming process of the streaming client to the running process of the scene rendering client through the scene rendering client, wherein the streaming process is used to obtain the target screen frame rendered by the scene rendering client, and forward the target screen frame to the running process of the streaming client.

[0146] Optionally, the forwarding unit is used to: forward the received operation data to the streaming process through the running process of the streaming client; forward the operation data to the running process of the scene rendering client through the streaming process, wherein the scene rendering client is used to forward the operation data to the target application.

[0147] Optionally, the data streaming device is also used for: when the target server is configured with multiple graphics cards, running the target number of virtual machines on the target server according to the target number of graphics cards configured by the target server, wherein the virtual machines are matched one-to-one with the graphics cards; and running one of the target applications and the streaming client on each of the multiple virtual machines, wherein each of the virtual machines is connected to one of the virtual scene display terminals.

[0148] Optionally, the data streaming device is used to: run a target development application when the target application is in an editing state, wherein the target development application is used to edit the target application; and run the target application in a preview state according to the target development application.

[0149] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0150] According to another aspect of the embodiments of the present invention, an electronic device for implementing the above-mentioned data streaming method is also provided. The electronic device may be Fig.13 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.13 As shown, the electronic device includes a memory 1302 and a processor 1304. The memory 1302 stores a computer program, and the processor 1304 is configured to execute the steps in any of the above method embodiments through the computer program.

[0151] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0152] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0153] S1, when a target server is connected to a virtual scene display terminal, in response to an application running request sent by the virtual scene display terminal, running a target application and a streaming client matching the target application on the target server;

[0154] S2, when receiving the operation data sent by the virtual scene display terminal, forwarding the operation data to the target application through the streaming client, wherein the operation data is used to control the virtual character in the target application;

[0155] S3, upon receiving the scene data sent by the target application, rendering the virtual scene in the target application according to the scene data to obtain a target screen frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located;

[0156] S4, pushing the encoded data of the target picture frame to the virtual scene display terminal through the streaming client.

[0157] Alternatively, a person skilled in the art may understand that: Fig.13The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig.13 The electronic device and the electronic equipment described above are not limited in structure. Fig.13 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.13 Different configurations are shown.

[0158] Among them, the memory 1302 can be used to store software programs and modules, such as program instructions / modules corresponding to the data streaming method and device in the embodiment of the present invention. The processor 1304 executes various functional applications and data streaming by running the software programs and modules stored in the memory 1302, that is, to implement the above-mentioned data streaming method. The memory 1302 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1302 may further include a memory remotely located relative to the processor 1304, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1302 may be specifically, but not limited to, used to store information such as various elements in the scene screen, data acquisition information, and the like. As an example, such as Fig.13 As shown, the memory 1302 may include, but is not limited to, the running unit 1202, forwarding unit 1204, rendering unit 1206, and streaming unit 1208 in the data streaming device. In addition, other module units in the data streaming device may also be included but are not limited to, which will not be repeated in this example.

[0159] Optionally, the transmission device 1306 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 1306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1306 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0160] In addition, the electronic device further includes: a display 1308 for displaying a virtual scene in an interface; and a connection bus 1310 for connecting various module components in the electronic device.

[0161] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, a peer-to-peer network may be formed between the nodes, and any form of computing device, such as a server, a terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0162] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0163] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0164] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned data acquisition method.

[0165] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0166] S1, when a target server is connected to a virtual scene display terminal, in response to an application running request sent by the virtual scene display terminal, running a target application and a streaming client matching the target application on the target server;

[0167] S2, when receiving the operation data sent by the virtual scene display terminal, forwarding the operation data to the target application through the streaming client, wherein the operation data is used to control the virtual character in the target application;

[0168] S3, upon receiving the scene data sent by the target application, rendering the virtual scene in the target application according to the scene data to obtain a target screen frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located;

[0169] S4, pushing the encoded data of the target picture frame to the virtual scene display terminal through the streaming client.

[0170] According to another aspect of the embodiments of the present invention, a virtual scene display device for implementing the above virtual scene display method is also provided. Fig.14 As shown, the device comprises:

[0171] An operating unit 1402 is used for a data receiving client running on a virtual scene display terminal;

[0172] The receiving unit 1404 is used to receive the encoded data sent by the streaming client running on the target server through the data receiving client when the virtual scene display terminal is connected to the target server;

[0173] A display unit 1406, configured to display the virtual scene where the virtual character controlled by the current object in the target application is located according to the decoding result of the encoded data;

[0174] The control unit 1408 is used to forward the operation data to the streaming client through the data receiving client when the virtual scene display terminal receives the operation data, wherein the operation data is used to control the virtual character in the target application.

[0175] Optionally, the display device of the above-mentioned virtual scene is also used to: in response to a selection operation of a target area from multiple candidate areas, send an establishment request to a server cluster corresponding to the target area; when the server cluster corresponding to the target area includes a target server in an idle state, establish a connection with a streaming client currently running on the target server through a data receiving client.

[0176] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0177] According to another aspect of the embodiments of the present invention, an electronic device for implementing the above-mentioned data streaming method is also provided. The electronic device may be Fig.15 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.15As shown, the electronic device includes a memory 1502 and a processor 1504. The memory 1502 stores a computer program, and the processor 1504 is configured to execute the steps in any of the above method embodiments through the computer program.

[0178] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0179] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0180] S1, a data receiving client running on a virtual scene display terminal;

[0181] S2, when the virtual scene display terminal is connected to the target server, receiving the encoded data sent by the streaming client running on the target server through the data receiving client;

[0182] S3, displaying the virtual scene where the virtual character controlled by the current object in the target application is located according to the decoding result of the encoded data;

[0183] S4, when the virtual scene display terminal receives the operation data, the operation data is forwarded to the streaming client through the data receiving client, wherein the operation data is used to control the virtual character in the target application.

[0184] Alternatively, a person skilled in the art may understand that: Fig.15 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig.15 The electronic device and the electronic equipment described above are not limited in structure. Fig.15 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.15 Different configurations are shown.

[0185] Among them, the memory 1502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data streaming method and device in the embodiments of the present invention. The processor 1504 executes various functional applications and data streaming by running the software programs and modules stored in the memory 1502, that is, to implement the above-mentioned data streaming method. The memory 1502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1502 may further include a memory remotely located relative to the processor 1504, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1502 may be specifically, but not limited to, used to store information such as various elements in the scene screen, data acquisition information, and the like. As an example, such as Fig.15 As shown, the memory 1502 may include, but is not limited to, the operation unit 1402, the receiving unit 1404, the display unit 1406, and the control unit 1408 in the display device of the virtual scene. In addition, other module units in the display device of the virtual scene may also be included but are not limited to, which will not be repeated in this example.

[0186] Optionally, the transmission device 1506 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1506 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0187] In addition, the electronic device further includes: a display 1508 for displaying a virtual scene in an interface; and a connection bus 1510 for connecting various module components in the electronic device.

[0188] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, a peer-to-peer network may be formed between the nodes, and any form of computing device, such as a server, a terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0189] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0190] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0191] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned data acquisition method.

[0192] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0193] S1, a data receiving client running on a virtual scene display terminal;

[0194] S2, when the virtual scene display terminal is connected to the target server, receiving the encoded data sent by the streaming client running on the target server through the data receiving client;

[0195] S3, displaying the virtual scene where the virtual character controlled by the current object in the target application is located according to the decoding result of the encoded data;

[0196] S4, when the virtual scene display terminal receives the operation data, the operation data is forwarded to the streaming client through the data receiving client, wherein the operation data is used to control the virtual character in the target application.

[0197] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0198] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the above methods of various embodiments of the present invention.

[0199] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0200] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0201] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0203] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data streaming method, characterized in that: include: When a target server is connected to a virtual scene display terminal, in response to an application running request sent by the virtual scene display terminal, a target application and a streaming client matching the target application are run on the target server; When the operation data sent by the virtual scene display terminal is received, the operation data is forwarded to the target application through the streaming client, wherein the operation data is used to control the virtual character in the target application; Upon receiving the scene data sent by the target application, rendering the virtual scene in the target application according to the scene data to obtain a target screen frame, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located; The encoded data of the target picture frame is pushed to the virtual scene display terminal through the streaming client.

2. The method according to claim 1, characterized in that In the case of receiving the scene data sent by the target application, rendering the virtual scene in the target application according to the scene data to obtain the target picture frame also includes: When the target application matches the first streaming mode, in response to a virtual device creation request sent by the target application, a virtual hardware device is created through the streaming client, wherein the virtual hardware device is used to provide a rendering interface; In response to a canvas creation request sent by the target application, a virtual canvas is created by the streaming client, wherein the virtual canvas is used to render a virtual scene in the target application based on the scene data through the rendering interface.

3. The method according to claim 2, characterized in that When the scene data sent by the target application is received, rendering the virtual scene in the target application according to the scene data to obtain a target picture frame includes: In response to a first rendering instruction sent by the target application, sending a canvas identifier of the virtual canvas to the target application through the streaming client, wherein the first rendering instruction is used to obtain the canvas identifier of the virtual canvas in an idle state; receiving the scene data sent by the target application, and performing scene rendering on the virtual scene in the target application based on the scene data on the virtual canvas indicated by the canvas identifier; Upon receiving a second rendering instruction sent by the target application, and in a case where the second rendering instruction indicates that the scene rendering has been completed, determining the target screen frame according to the virtual canvas.

4. The method according to claim 2, characterized in that: When the scene data sent by the target application is received, the virtual scene in the target application is rendered by the streaming client according to the scene data to obtain a target picture frame, further comprising: When a plurality of the virtual canvases are created by the streaming client, and a first canvas among the plurality of the virtual canvases is used to render the target picture frame, obtaining a reference picture frame rendered by a second canvas; Video encoding is performed according to the reference picture frame, and the encoded data of the reference picture frame is pushed to the virtual scene display terminal through the streaming client.

5. The method according to claim 2, characterized in that: When the operation data sent by the virtual scene display terminal is received, forwarding the operation data to the target application through the streaming client includes at least one of the following: In the case of acquiring the perspective operation data sent by the virtual scene display terminal, forwarding the perspective data to the target application in response to the perspective data acquisition request sent by the target application, wherein the perspective operation data is used to indicate the object perspective matching the current terminal posture of the virtual scene display terminal; In the case of acquiring the position operation data sent by the virtual scene display terminal, forwarding the position operation data to the target application in response to the position data acquisition request sent by the target application, wherein the position operation data is used to indicate the position of the object matching the current spatial position of the virtual scene display terminal; When the action operation data sent by the virtual scene display terminal is obtained, the action operation data is forwarded to the target application in response to the action data acquisition request sent by the target application, wherein the action operation data is used to indicate the object action detected by the virtual scene display terminal.

6. The method according to claim 1, characterized in that In the case of receiving the scene data sent by the target application, rendering the virtual scene in the target application according to the scene data by the streaming client to obtain the target picture frame also includes: When the target application matches the second streaming mode, a scene rendering client is run on the target server, wherein the scene rendering client is used to render a virtual scene in the target application according to the scene data; The streaming process of the streaming client is added to the running process of the scene rendering client through the scene rendering client, wherein the streaming process is used to obtain the target screen frame rendered by the scene rendering client and forward the target screen frame to the running process of the streaming client.

7. The method according to claim 6, characterized in that When the operation data sent by the virtual scene display terminal is received, forwarding the operation data to the target application through the streaming client includes: Forwarding the received operation data to the streaming process through the running process of the streaming client; The operation data is forwarded to the running process of the scene rendering client through the streaming process, wherein the scene rendering client is used to forward the operation data to the target application.

8. The method according to claim 1, characterized in that In the case where the target server is connected to the virtual scene display terminal, in response to an application running request sent by the virtual scene display terminal, before running the target application and the streaming client matching the target application on the target server, the method further includes: In the case where the target server is configured with multiple graphics cards, running the target number of virtual machines on the target server according to the target number of the graphics cards configured on the target server, wherein the virtual machines are matched one-to-one with the graphics cards; One of the target applications and the streaming client is run on each of the multiple virtual machines, wherein each of the virtual machines is connected to one of the virtual scene display terminals.

9. The method according to claim 1, characterized in that: In the case where the target server is connected to the virtual scene display terminal, in response to an application running request sent by the virtual scene display terminal, running a target application and a streaming client matching the target application on the target server includes: When the target application is in an editing state, running a target development application, wherein the target development application is used to perform application editing on the target application; The target application in the preview state is run according to the target development application.

10. A method for displaying a virtual scene, characterized in that: include: A data receiving client running on a virtual scene display terminal; When the virtual scene display terminal is connected to the target server, the data receiving client receives the encoded data sent by the streaming client running on the target server; Displaying the virtual scene where the virtual character controlled by the current object in the target application is located according to the decoding result of the encoded data; When the virtual scene display terminal receives the operation data, the operation data is forwarded to the streaming client through the data receiving client, wherein the operation data is used to control the virtual character in the target application.

11. The method according to claim 10, characterized in that In the case where the virtual scene display terminal is connected to the target server, before the data receiving client receives the encoded data sent by the streaming client running on the target server, the method further includes: In response to a selection operation of a target area from a plurality of candidate areas, sending a creation request to a server cluster corresponding to the target area; In the case that the server cluster corresponding to the target area includes the target server in an idle state, a connection is established between the data receiving client and the streaming client currently running on the target server.

12. A data streaming device, characterized in that: include: An operation unit, configured to operate a target application and a streaming client matching the target application on the target server in response to an application operation request sent by the virtual scene display terminal when a connection is established between the target server and the virtual scene display terminal; A forwarding unit, configured to forward the operation data sent by the virtual scene display terminal to the target application through the streaming client upon receiving the operation data, wherein the operation data is used to control the virtual character in the target application; A rendering unit, configured to render a virtual scene in the target application according to the scene data to obtain a target screen frame upon receiving the scene data sent by the target application, wherein the scene data is used to indicate the virtual scene in which the virtual character is currently located; A streaming unit is used to push the encoded data of the target picture frame to the virtual scene display terminal through the streaming client.

13. A virtual scene display device, characterized in that: include: An operating unit, used for a data receiving client running on a virtual scene display terminal; A receiving unit, configured to receive, through the data receiving client, coded data sent by a streaming client running on the target server when the virtual scene display terminal is connected to the target server; A display unit, configured to display a virtual scene in which a virtual character controlled by a current object in a target application is located according to a decoding result of the encoded data; A control unit is used to forward the operation data to the streaming client through the data receiving client when the virtual scene display terminal receives the operation data, wherein the operation data is used to control the virtual character in the target application.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 9 or 10 to 11 when executed.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 9 or 10 to 11 are implemented.

16. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method described in any one of claims 1 to 9 or 10 to 11 through the computer program.

Images