Display method of cloud terminal product

Abstract

The invention discloses a cloud terminal product display method, and belongs to the technical field of videos. The method comprises the steps of determining a form of a cloud terminal product and a rendering attribute corresponding to the cloud terminal product according to scene information of a pre-acquired virtual scene; generating rendering data of the cloud terminal product displayed in the virtual scene according to the form of the cloud terminal product and the rendering attribute corresponding to the cloud terminal product; and sending the rendering data to a client.

Description

Technical Field

[0001] The present application belongs to the field of video technology, and specifically relates to a method for displaying a cloud terminal product. Background Art

[0002] In the related art, if users need to use cloud terminal products in products such as augmented reality (AR) / virtual reality (VR), there are two main options: (1) use the browser in the operating system of AR / VR products to access cloud terminal products; (2) install APP in AR / VR products and access cloud terminal products in APP mode. Users have a poor experience using cloud terminal products in virtual scenes. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method for displaying a cloud terminal product, which can solve the problem that users have a poor experience when using the cloud terminal product in a virtual scene.

[0004] In a first aspect, an embodiment of the present application provides a method for displaying a cloud terminal product, which is applied to a server, and the method includes: determining the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product based on pre-acquired scene information of a virtual scene; generating rendering data for displaying the cloud terminal product in the virtual scene based on the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product; and sending the rendering data to a client.

[0005] In a second aspect, an embodiment of the present application provides another method for displaying a cloud terminal product, which is applied to a client, and the method includes: sending scene identification information of a virtual scene to a server, wherein the scene identification information is used to determine the form of the cloud terminal product; receiving rendering data corresponding to the form of the cloud terminal product sent by the server; and displaying the rendering data in the virtual scene.

[0006] In a third aspect, an embodiment of the present application provides a display device for a cloud terminal product, which is applied to a server, and the device includes: a determination module, which is used to determine the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product based on pre-acquired scene information of the virtual scene; a generation module, which is used to generate rendering data for displaying the cloud terminal product in the virtual scene based on the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product; and a transmission module, which is used to send the rendering data to a client.

[0007] In a fourth aspect, an embodiment of the present application provides another display device for a cloud terminal product, which is applied to a client, and the device includes: a sending module, which is used to send scene identification information of a virtual scene to a server, wherein the scene identification information is used to determine the form of the cloud terminal product; a receiving module, which is used to receive rendering data corresponding to the form of the cloud terminal product sent by the server; and a display module, which is used to display the rendering data in the virtual scene.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0010] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect or the second aspect.

[0011] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect or the second aspect.

[0012] In the embodiment of the present application, the server can determine the form of the cloud terminal product and the corresponding rendering attributes of the cloud terminal product according to the pre-acquired scene information of the virtual scene; generate rendering data displayed by the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the corresponding rendering attributes of the cloud terminal product; and send the rendering data to the client. This can not only make the user's experience of using the cloud terminal product in the virtual scene more adaptive and convenient, but also enhance the user's sense of immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram showing a flow chart of a method for displaying a cloud terminal product provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram showing a scene recognition provided by an embodiment of the present application;

[0015] Figure 3 A schematic diagram of processing a video stream provided by an embodiment of the present application is shown;

[0016] Figure 4 A schematic diagram showing a dynamic rendering provided by an embodiment of the present application is shown;

[0017] Figure 5 A schematic diagram showing an interactive processing and interface rendering provided by an embodiment of the present application;

[0018] Figure 6 A schematic diagram showing a flow chart of another message sending method provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram showing the structure of a display device for a cloud terminal product provided in an embodiment of the present application is shown;

[0020] Figure 8 A schematic diagram showing the structure of another display device for a cloud terminal product provided in an embodiment of the present application;

[0021] Fig. 9 A structural block diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] When using cloud terminals, more computing power can be migrated to the cloud. For example, tasks with heavy computing power, such as computing, storage, and control, are placed in the cloud for processing. The cloud determines the display data of the cloud terminal, and the cloud terminal only plays a role in connection and display. After the cloud terminal is connected to the cloud server through the protocol, it can achieve the same usage effect as an ordinary terminal, meeting the user's various needs such as office, life, and entertainment.

[0025] At present, cloud terminal products are mainly provided to users in the form of applications (Application, APP), mini-programs, HTML5 (Hypertext Markup Language (HTML) is an extended markup language widely used on the Internet and the main language for web documents. HTML5 is a new version of HTML, referred to as H5) web pages. If users need to use cloud phones / cloud terminal products in augmented reality / products, there are two main options:

[0026] (1) Use the browser in the AR / VR operating system to access cloud terminal products.

[0027] (2) Install the APP in the AR / VR product and access the cloud terminal product via the APP.

[0028] The related technical solutions have the following disadvantages:

[0029] (1) It is not convenient to use a browser to access cloud terminal products in an AR / VR operating system. Users need to manually open the browser and enter the URL in a virtual environment, which increases the complexity of the user's operation. At the same time, the browser access method is less immersive.

[0030] (II) Currently, users use browsers or APPs to access cloud terminal products. The resolution, user interface (UI) layout and product form of cloud terminal products are fixed, and users' experience in virtual scenes is relatively poor.

[0031] The following is a detailed description of the cloud terminal product display method provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0032] Figure 1 A method for displaying a cloud terminal product provided by an exemplary embodiment of the present application is shown, which is applied to a server. The method 100 can be executed by an electronic device, which can be a cloud terminal or other device. Figure 1 As shown, the method mainly includes the following steps:

[0033] S101: Determine the form of a cloud terminal product and rendering attributes corresponding to the cloud terminal product according to pre-acquired scene information of a virtual scene.

[0034] In an embodiment of the present application, the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product can be determined based on the pre-acquired scene information of the virtual scene, so that the form of the cloud terminal product can be determined according to the actual situation of the virtual scene, which can provide users with a more immersive experience. In an embodiment of the present application, the form of the cloud terminal product can be a mobile phone, a tablet, a computer, or other forms, which are not specifically limited in the embodiment of the present application. The rendering attributes corresponding to the form of the cloud terminal product include screen ratio, resolution, typical values ​​of rendering size, etc., which are not specifically limited in the embodiment of the present application. Taking mobile phones, tablets, and computers as examples, the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product are shown in Table 1, and a total of three major categories of P①②③ / T①② / C①②③ and 8 subcategories are determined. In the specific implementation method, it can be determined according to the actual situation.

[0035] Table 1

[0036]

[0037] In an optional implementation, determining the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product according to the pre-acquired scene information of the virtual scene includes:

[0038] The scene information is identified based on a pre-trained scene classification model to determine the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product.

[0039] In the embodiment of the present application, a pre-trained scene classification model can be used to identify scene information, thereby determining the form of the cloud terminal product and the corresponding rendering attributes of the cloud terminal product. The scene classification model can improve the speed of scene recognition and increase the degree of user immersion. Figure 2 As shown, a machine learning algorithm can be used to perform recognition training on key scene photos such as "holding a mobile phone with one hand", "holding a tablet with two hands", and "desktop office". The scene recognition process may include the following steps:

[0040] Step 1: Data collection. Use the environmental sensors of AR / VR hardware, including camera images, gyroscopes, accelerometers, etc. to collect environmental data. These data will form a rich scene dataset for training scene classification models.

[0041] Step 2: Train the scene classification model. Use machine learning algorithms to train the collected data to develop an intelligent scene classification model. This model will learn to recognize different intelligent scenes, such as whether the user is holding a mobile phone with one hand, holding a tablet with two hands, or working on a desktop. During the model training process, images and environmental data of various devices of different forms will be included to ensure that the model has a wide range of scene recognition capabilities.

[0042] Step 3: Model deployment: The trained scene classification model will be deployed to the cloud terminal device (server) for real-time scene recognition.

[0043] Step 4: Real-time scene recognition. When the user starts the scene recognition module on the cloud phone / cloud terminal device, the cloud terminal will quickly recognize the current scene. The scene information will be passed to the rendering system so that corresponding rendering adjustments can be made based on the recognition results.

[0044] For those who do not choose to enable the Scene Recognition Module, the desired scene mode can be manually selected, thereby activating the corresponding rendering settings and passing the scene information to the rendering system.

[0045] In an embodiment of the present application, assuming that the virtual scene identified according to the scene classification model is a single-handed mobile phone, it can be determined that the form of the cloud terminal product required in the virtual scene is a mobile phone. At the same time, various properties of the mobile phone, such as screen ratio, size, etc., can be determined based on the corresponding rendering properties of the cloud terminal product required by the virtual scene, so as to provide an immersive experience for the user.

[0046] In an optional implementation, after determining the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product according to the pre-acquired scene information of the virtual scene, the method further includes:

[0047] According to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product, computing resources corresponding to the cloud terminal product are determined from existing computing resources.

[0048] In the embodiment of the present application, after determining the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product, the computing resources corresponding to the form of the cloud terminal product can be determined from the existing computing resources to generate rendering data corresponding to the cloud terminal product. That is, according to the scene recognition results, the resource configuration of the cloud terminal is dynamically adjusted to provide the best performance and user experience.

[0049] In actual applications, it can include two steps, first allocation and then actual determination. Step 1: The server can dynamically allocate computing resources to meet user needs based on the recognition results of the current scene. This includes resource specifications such as the number of central processing unit (CPU) cores, memory capacity, and the number of virtual graphics processing units (vGPU). Users can choose the required configuration when ordering to ensure the best performance and user experience in different scenarios. For example, the following Table 2:

[0050] Table 2

[0051] Product form CPU(core) Memory(G) vGPUs cell phone 4 8 2 flat 4 8 2 computer 8 16 4

[0052] Step 2: Dynamic resource allocation: Determine the startup form and resource specifications based on the scene conditions transmitted by scene recognition. Compared with the static resource configuration in the related art, the embodiment of the present application can dynamically allocate computing resources according to scene requirements to maximize performance and efficiency.

[0053] In the embodiment of the present application, the resolution in the rendering attribute can be determined according to the network status. In the related technology, the server can dynamically adjust the resolution according to the user's network status, such as from 720P@30FPS to 1080P@FPS, where FPS is the abbreviation of Frames Per Second, which is the number of frames transmitted per second. 1080p is a display format that reaches a resolution of 1920×1080 under progressive scanning, and P means progressive scanning. The embodiment of the present application can realize dynamic switching according to the form and layout of the scene on the basis of traditional dynamic resolution adjustment, and can also realize dynamic adjustment of resolution according to network conditions, realizing a set of multi-state compatibility of computing power.

[0054] S102: Generate rendering data for displaying the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product.

[0055] In the embodiment of the present application, the rendering data displayed by the cloud terminal product in the virtual scene can be generated according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product. Compared with the fixed rendering attributes (resolution and interface layout, etc.) in the related art, the embodiment of the present application can generate rendering data according to the form of different cloud terminal products and the rendering attributes corresponding to the cloud terminal products, so that the cloud terminal product can be flexibly displayed in different forms in the virtual scene, which can provide users with a more adaptive and convenient experience.

[0056] In an optional implementation, generating rendering data displayed by the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product may include the following steps:

[0057] Step 1, according to the form of the cloud terminal product and the rendering properties corresponding to the form of the cloud terminal product, call the corresponding pre-configured resource file; in the embodiment of the present application, in order to support different modes, the resource file can be optimized in advance, and the resource file includes images, icons and text resources, etc., to adapt to different resolutions. These optimized resource files can provide clearer and more attractive interface elements at different resolutions. When generating rendering data, the pre-configured resource file can be directly called to improve efficiency.

[0058] Step 2, according to the rendering properties corresponding to the form of the cloud terminal product and the resource file, generate the rendering data corresponding to the form of the cloud terminal product, wherein the rendering data includes interface element data. In order to better support the forms of different cloud terminal products, the code logic of automatic adaptation layout and implementation of user interface elements can be performed. According to the rendering properties and the loaded layout file, the size, position and number of interface elements will be automatically adjusted to adapt to different modes. In an embodiment of the present application, corresponding interface element data can be generated according to the rendering properties and related resource files to provide users with an immersive experience.

[0059] In an optional implementation, the rendering data includes video frame data, and generating the rendering data displayed by the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product may include the following steps:

[0060] Step 1, obtain the two-dimensional 2D video stream related to the cloud terminal product and compress the 2D video stream; in the process of generating video frame data, image acquisition and preprocessing can be performed first, and the 2D video stream output by the cloud terminal operating system can be acquired. For example, the acquisition end can establish a connection with the operating system of the cloud terminal through Virtual Network Computing (VNC) to achieve access to the graphical interface. The display resolution and ratio of the cloud terminal are dynamically set through the VNC connection to obtain the 2D video stream. Then the video stream is preprocessed (the video stream is compressed), and the Fast Forward Moving Picture Experts Group (ffmpeg) multimedia encoding tool can be used to compress the video data into a format suitable for transmission, such as H.264 encoding. In the embodiment of the present application, other methods can also be used to compress the video stream, and the transmission format can also use other formats, which are not specifically limited in the embodiment of the present application.

[0061] Step 2, receiving the surface parameters sent by the client; wherein the surface parameters are used to determine the projection matrix and the view matrix. In an embodiment of the present application, the surface parameters may include relevant parameters in the projection matrix and the view matrix, such as the field of view angle, the distance between the near clipping plane and the far clipping plane, the position of the human eye, etc.

[0062] Step 3, according to the form of the cloud terminal product, the rendering properties corresponding to the cloud terminal product and the surface parameters, each frame image in the 2D video stream is transformed using a perspective matrix to generate the curved video frame data, wherein the perspective matrix is ​​obtained by calculating the projection matrix and the view matrix. In an embodiment of the present application, a perspective matrix can be used to perform a perspective transformation on the image of each frame to achieve the surface transformation of the 2D image. This separation method allows the rendering engine to more easily modify the projection matrix and the view matrix to achieve various effects, such as perspective projection, orthogonal projection, camera movement, rotation, etc., without having to recalculate the entire transformation, which helps to improve the maintainability and flexibility of the rendering pipeline.

[0063] In the specific implementation, you can first create a projection matrix perspective_matrix: use perspective projection to simulate the field of view of the human eye in order to present realistic images on the surface. There are three parameters in total:

[0064] (1) FOV: Field of view angle, which indicates the size of the visible area.

[0065] (2) aspect_ratio: the aspect ratio of the screen (16.0 / 9.0 in the case of 16:9). It should be noted that the screen aspect ratios in the corresponding rendering attributes are different for different cloud terminal products. As shown in Table 1 above, the screen aspect ratios of mobile phones are 9:16, tablets are 4:3, and computers are 16:9.

[0066] (3)near and far: the distance between the near clipping plane and the far clipping plane.

[0067]

[0068] Then create the view matrix view_matrix: The view matrix defines the position and direction of the human eye to determine the viewing angle of the scene. The curved screen is not curved up and down, but only curved left and right. The overall calculation is as follows, with a total of three parameters:

[0069] (1)eye: the location of the human eye.

[0070] (2) Center: The target point observed by the human eye.

[0071] (3) up: the upward direction of the human eye.

[0072]

[0073] By calculating the projection matrix and the view matrix separately through the above two processes, you can better control and manage the camera and rendering effects. Ultimately, the scene needs to be mapped from the world coordinate system to the coordinate system of the AR / VR screen. By multiplying the projection matrix and the view matrix, the three-dimensional (3D) scene can be correctly mapped to the screen in a single transformation step. Multiply the projection matrix and the view matrix to obtain the perspective matrix perspective_view_matrix:

[0074] perspective_view_matrix=perspective_matrix*view_matrix

[0075] In the embodiments of the present application, Figure 3 As shown in the figure, the server is responsible for running the actual cloud terminal operating system and is managed in a containerized manner. At the transmission level, it abstracts the operating system into instruction streams and video streams. The server can receive dynamic requests from the client and present the operating system's user interface in the form of a video stream to meet different form requirements.

[0076] In an optional implementation, after sending the rendering data to the client, the method further includes: receiving head tracking data, eye tracking data, and hand tracking data sent by the client;

[0077] Based on the head tracking data, the eye tracking data and the hand tracking data, dynamically render the cloud terminal product to obtain dynamic rendering data;

[0078] The dynamic rendering data is sent to the client.

[0079] In the above optional implementation, the step of dynamically rendering the cloud terminal product based on the head tracking data, the eye tracking data, and the hand tracking data to obtain dynamic rendering data may include the following steps:

[0080] Step 1: Obtain the user's head position and viewpoint based on the head tracking data and the eye tracking data; the server can use the head tracking data to obtain the user's head movement, including direction, rotation and tilt, and then continuously update the position and direction of the cloud terminal product to ensure that the user's perspective in the virtual environment is consistent with the head movement. The eye tracking data can be used to determine the user's gaze point, thereby realizing the functions of line of sight interaction and gaze triggering.

[0081] Step 2, obtaining the user's hand position and hand movements based on the hand tracking data; the user's hand position and movement can be obtained through the hand tracking data, and the hand movement includes the user's gestures.

[0082] Step 3, according to the user's head position and viewpoint, the user's hand position and hand movement, based on the pre-acquired simulation model, adjust the rendering of the cloud terminal product to obtain dynamic rendering data. Figure 4 As shown, a simulation model that matches the real environment can be rendered, using the sensors of the AR / VR device (depth sensor, color data) to measure or track the physical position of objects such as the user's palm or desk. Then, based on the measurement data of the real-world objects, the 3D model and interface of the cloud terminal are rendered to the viewpoint and position of the virtual camera to match the real environment. The simulation model is rendered according to the actual size of the physical world. In order to enable user interaction with the cloud terminal product, the cloud terminal product can be updated accordingly through the user's head position and viewpoint, the user's hand position and hand movements, and the rendering of the virtual scene can be adjusted in real time. Dynamic rendering data may include parameters such as the position, size, angle or transparency of the rendered object (cloud terminal product) to reflect the user's interaction and actions, thereby creating a more immersive and realistic virtual experience, allowing users to obtain a simulated cloud terminal product at the visual level. As shown Figure 5 As shown, dynamic rendering can be used to enable interaction between users and cloud terminal products.

[0083] For example, through head tracking data, eye tracking data and hand tracking data, the user's actions are obtained to change the position of the cloud terminal product (such as a cloud computer) for the user. For example, according to the simulation model, it is determined that the cloud computer moves from the user's hand to the table, then the position of the cloud computer needs to be updated and the generated dynamic rendering data is sent to the client so that the user can see the cloud computer on the table in the virtual scene.

[0084] S103: Send the rendering data to the client.

[0085] In an embodiment of the present application, the server can send the generated rendering data to the client so that the cloud terminal product can be displayed in a virtual scene, which can not only improve the applicability of the cloud terminal product, but also make the user experience more convenient and bring a better user experience.

[0086] The server can determine the form of the cloud terminal product and the corresponding rendering attributes of the cloud terminal product based on the pre-acquired scene information of the virtual scene; generate rendering data for the cloud terminal product to be displayed in the virtual scene based on the form of the cloud terminal product and the corresponding rendering attributes of the cloud terminal product; and send the rendering data to the client. This can not only make the user's experience of using the cloud terminal product in the virtual scene more adaptive and convenient, but also enhance the user's sense of immersion.

[0087] Figure 6 Another method for displaying cloud terminal products provided by an exemplary embodiment of the present application is shown, which is applied to a client. The method 600 can be executed by an electronic device, which can be an XR handle, an AR helmet, and the like. Figure 6 As shown, the method mainly includes the following steps:

[0088] S601: Send scene identification information of the virtual scene to the server.

[0089] The scene recognition information is used to determine the form of the cloud terminal product.

[0090] In an embodiment of the present application, the client can send scene recognition information of the virtual scene to the server, so that the server can determine the form of the cloud terminal product and enhance the user's immersion. In actual applications, the scene recognition information sent by the client can be a specific scene of the virtual scene, and the server can perform scene recognition to determine the form of the cloud terminal product. The client can run inside the AR / VR computing device, use local computing power to directly implement scene recognition, determine the form that the cloud terminal product needs to present, including mobile phones, tablets, computers, etc., and then send the scene recognition information to the server.

[0091] S602: Receive rendering data corresponding to the form of the cloud terminal product sent by the server.

[0092] In an embodiment of the present application, the client can receive rendering data corresponding to the form of the cloud terminal product sent by the server, which can not only save the client's memory but also improve efficiency.

[0093] In an optional implementation, after sending the scene identification information of the virtual scene to the server, the method further includes:

[0094] The pre-acquired surface parameters are sent to the server, wherein the surface parameters are used to determine a projection matrix and a view matrix.

[0095] In an embodiment of the present application, the client can send surface parameters for generating video frame data to the server, the surface parameters including relevant parameters of the projection matrix and the view matrix, such as the field of view angle, the distance between the near clipping plane and the far clipping plane, the position of the human eye, and other parameters, so that the server can curve the video stream.

[0096] S603: Display the rendering data in the virtual scene.

[0097] In the embodiment of the present application, the client can render the 2D / 3D model of the cloud terminal product in real time, and then display the cloud terminal product in the virtual scene. Generally, the rendering data corresponding to the cloud terminal can be combined with the original display data of the virtual scene by the image engine to display the cloud terminal, or the image engine can be used to directly display and process the rendering data. In actual applications, other methods can also be used to display the rendering data.

[0098] In practical applications, when the rendering data is video frame data, the client can leave enough pixel area in the cloud terminal product model and embed the video frame data into the model. In the embodiment of the present application, the video frame data is obtained by the projection matrix and view matrix composed of the surface parameters and the rendering attributes of the cloud terminal product. The projection matrix and view matrix are used to realize the surface conversion of the 2D video stream image, which can better utilize the advantage of the large field of view of AR / VR products and bring better experience to users.

[0099] In an optional implementation, after displaying the rendering data in the virtual scene, the method further includes the following steps:

[0100] Step 1, sending the acquired head tracking data, eye tracking data and hand tracking data to the server; in an embodiment of the present application, the client can call the built-in sensor of the AR / VR device to monitor the movement of the head, including direction, rotation and tilt, to obtain head tracking data. At the same time, eye tracking technology is used to track the movement and gaze point of the user's eyes to obtain eye tracking data. The client can transmit the head and eye tracking information to the server to achieve real-time monitoring of the head and eyes. The client can also use a hand tracking device or camera to monitor the position, movement and gesture of the hand to obtain hand tracking data, and then transmit the hand tracking data to the server to reflect the user's interaction and action.

[0101] Step 2, receiving dynamic rendering data sent by the server; the client can receive dynamic rendering data sent by the server, and the dynamic rendering data is data updated by the server on the position, size, angle or transparency of the cloud terminal product according to the head tracking data, eye tracking data and hand tracking data, which can reflect the user's interaction and action.

[0102] Step 3: Display the dynamic rendering data in the virtual scene. Based on the dynamic rendering data, the client can display the dynamic rendering data in the virtual scene through the image engine, thereby creating a more immersive and realistic virtual experience, so that the user can obtain a simulated cloud terminal device at the visual level.

[0103] In the embodiment of the present application, the client can send scene recognition information for determining the form of the cloud terminal product to the server, then receive rendering data corresponding to the form of the cloud terminal product sent by the server, and finally display the rendering data in the virtual scene. Not only can the cloud terminal product be displayed in a variety of forms such as mobile phones, tablets, and computers in virtual devices such as AR / VR, improving the applicability of the cloud terminal product, but also providing a better user experience.

[0104] The display method of the cloud terminal product provided in the embodiment of the present application can be executed by a display device of the cloud terminal product. The display method of the cloud terminal product is executed by a display device of the cloud terminal product as an example to illustrate the display device of the cloud terminal product provided in the embodiment of the present application.

[0105] Figure 7 The schematic diagram of the structure of the display device of the cloud terminal product provided by an exemplary embodiment of the present application is shown. The message display device can be applied to the server and can be implemented as follows: Figure 1 In the embodiment shown, all or part of the content, the display device of the cloud terminal product includes: a determination module 701, a generation module 702, and a transmission module 703.

[0106] In an embodiment of the present application, a determination module 701 is used to determine the form of a cloud terminal product and the rendering attributes corresponding to the cloud terminal product based on pre-acquired scene information of a virtual scene; a generation module 702 is used to generate rendering data displayed by the cloud terminal product in the virtual scene based on the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product; and a transmission module 703 is used to send the rendering data to a client.

[0107] In an optional implementation, when the determination module 701 is used to determine the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product according to the pre-acquired scene information of the virtual scene, it is specifically used to:

[0108] The scene information is identified based on a pre-trained scene classification model to determine the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product.

[0109] In an optional implementation, the determination module 701 is further configured to determine computing resources corresponding to the cloud terminal product from existing computing resources according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product.

[0110] In an optional implementation, when the generating module 702 is used to generate the rendering data displayed by the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product, it is specifically used to:

[0111] According to the form of the cloud terminal product and the rendering properties corresponding to the form of the cloud terminal product, calling the corresponding pre-configured resource file;

[0112] The rendering data corresponding to the form of the cloud terminal product is generated according to the rendering properties corresponding to the form of the cloud terminal product and the resource file, wherein the rendering data includes interface element data.

[0113] In an optional implementation, the rendering data includes video frame data, and the generating module 702, when used to generate the rendering data displayed by the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product, is specifically used to:

[0114] Acquire a two-dimensional 2D video stream related to the cloud terminal product and compress the 2D video stream;

[0115] Receiving surface parameters sent by the client, wherein the surface parameters are used to determine a projection matrix and a view matrix;

[0116] According to the shape of the cloud terminal product, the rendering properties corresponding to the cloud terminal product and the surface parameters, each frame image in the 2D video stream is transformed using a perspective matrix to generate the curved video frame data, wherein the perspective matrix is ​​calculated by the projection matrix and the view matrix.

[0117] In an optional implementation, the generation module 702 is also used to receive head tracking data, eye tracking data and hand tracking data sent by the client; based on the head tracking data, the eye tracking data and the hand tracking data, dynamically render the cloud terminal product to obtain dynamic rendering data; and send the dynamic rendering data to the client.

[0118] In the above optional implementation, the generating module 702 is used for dynamically rendering the cloud terminal product based on the head tracking data, the eye tracking data and the hand tracking data to obtain the dynamic rendering data, specifically for:

[0119] Obtaining the user's head position and viewpoint according to the head tracking data and the eye tracking data; and obtaining the user's hand position and hand movement according to the hand tracking data;

[0120] According to the user's head position and viewpoint, the user's hand position and hand movement, and based on the pre-acquired simulation model, the rendering of the cloud terminal product is adjusted to obtain dynamic rendering data.

[0121] The display device of the cloud terminal product in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a cloud terminal.

[0122] The message display device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0123] The display device of the cloud terminal product provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the illustrated method embodiment will not be described again here.

[0124] Another cloud terminal product display method provided in the embodiment of the present application can be executed by another cloud terminal product display device. In the embodiment of the present application, another cloud terminal product display method executed by another cloud terminal product display device is used as an example to illustrate another cloud terminal product display device provided in the embodiment of the present application.

[0125] Figure 8 A schematic diagram of the structure of another display device of a cloud terminal product provided by an exemplary embodiment of the present application is shown. The display device of the cloud terminal product is applied to a client and can be implemented as follows: Figure 6 In the embodiment shown, all or part of the content, the display device of the cloud terminal product includes: a sending module 801, a receiving module 802 and a display module 803.

[0126] In an embodiment of the present application, a sending module 801 is used to send scene identification information of a virtual scene to a server, wherein the scene identification information is used to determine the form of a cloud terminal product; a receiving module 802 is used to receive rendering data corresponding to the form of the cloud terminal product sent by the server; and a display module 803 is used to display the rendering data in the virtual scene.

[0127] In an optional implementation, the sending module 801 is further used to send the pre-acquired surface parameters to the server, wherein the surface parameters are used to determine the projection matrix and the view matrix.

[0128] In an optional implementation, the sending module 801 is also used to send the acquired head tracking data, eye tracking data and hand tracking data to the server; the receiving module 802 is also used to receive the dynamic rendering data sent by the server; the display module 803 is also used to display the dynamic rendering data in the virtual scene.

[0129] The display device of the cloud terminal product in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. For example, the electronic device can be an augmented reality (AR) or virtual reality (VR) device, which is not specifically limited in the embodiment of the present application.

[0130] The display device of the cloud terminal product provided in the embodiment of the present application can achieve Figure 6 To avoid repetition, the various processes implemented in the illustrated method embodiment will not be described again here.

[0131] Alternatively, if Fig. 9 As shown, the embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores a program or instruction that can be run on the processor 901, and when the program or instruction is executed by the processor 701, the above Figure 1 The display method of the cloud terminal product shown or Figure 6The various steps of the display method of the cloud terminal product shown can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0132] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0133] The embodiment of the present application also provides a computer-readable storage medium on which a program or instruction is stored, and when the program or instruction is executed by a processor, the above Figure 1 The display method of the cloud terminal product shown or Figure 6 The various processes of the display method of the cloud terminal product shown can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0134] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0135] The present application also provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above Figure 1 The display method of the cloud terminal product shown or Figure 6 The various processes of the display method of the cloud terminal product shown can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0136] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0137] The present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the above Figure 1 The display method of the cloud terminal product shown or Figure 6 The various processes of the display method of the cloud terminal product shown can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0138] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0140] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for displaying cloud terminal products, applied to a server, characterized in that: include: Determining the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product according to the pre-acquired scene information of the virtual scene; Generating rendering data displayed by the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product; The rendering data is sent to the client.

2. The method according to claim 1, characterized in that The determining, according to the pre-acquired scene information of the virtual scene, the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product includes: The scene information is identified based on a pre-trained scene classification model to determine the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product.

3. The method according to claim 1, characterized in that After determining the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product according to the pre-acquired scene information of the virtual scene, the method further includes: According to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product, computing resources corresponding to the cloud terminal product are determined from existing computing resources.

4. The method according to claim 1, characterized in that: The generating, according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product, rendering data displayed by the cloud terminal product in the virtual scene includes: According to the form of the cloud terminal product and the rendering properties corresponding to the form of the cloud terminal product, calling the corresponding pre-configured resource file; The rendering data corresponding to the form of the cloud terminal product is generated according to the rendering properties corresponding to the form of the cloud terminal product and the resource file, wherein the rendering data includes interface element data.

5. The method according to claim 1, characterized in that The rendering data includes video frame data, and generating rendering data displayed by the cloud terminal product in the virtual scene according to the form of the cloud terminal product and the rendering attributes corresponding to the cloud terminal product includes: Acquire a two-dimensional 2D video stream related to the cloud terminal product and compress the 2D video stream; Receiving surface parameters sent by the client, wherein the surface parameters are used to determine a projection matrix and a view matrix; According to the shape of the cloud terminal product, the rendering properties corresponding to the cloud terminal product and the surface parameters, each frame image in the 2D video stream is transformed using a perspective matrix to generate the curved video frame data, wherein the perspective matrix is ​​calculated by the projection matrix and the view matrix.

6. The method according to claim 1, characterized in that After sending the rendering data to the client, the method further includes: Receiving the head tracking data, eye tracking data and hand tracking data sent by the client; Based on the head tracking data, the eye tracking data and the hand tracking data, dynamically render the cloud terminal product to obtain dynamic rendering data; The dynamic rendering data is sent to the client.

7. The method according to claim 6, characterized in that The step of dynamically rendering the cloud terminal product based on the head tracking data, the eye tracking data, and the hand tracking data to obtain dynamic rendering data includes: Obtaining the user's head position and viewpoint according to the head tracking data and the eye tracking data; and obtaining the user's hand position and hand movement according to the hand tracking data; According to the user's head position and viewpoint, the user's hand position and hand movement, and based on the pre-acquired simulation model, the rendering of the cloud terminal product is adjusted to obtain dynamic rendering data.

8. A method for displaying cloud terminal products, applied to a client, characterized in that: include: Sending scene identification information of the virtual scene to the server, wherein the scene identification information is used to determine the form of the cloud terminal product; Receiving rendering data corresponding to the form of the cloud terminal product sent by the server; The rendering data is displayed in the virtual scene.

9. The method according to claim 8, characterized in that After sending the scene identification information of the virtual scene to the server, the method further includes: The pre-acquired surface parameters are sent to the server, wherein the surface parameters are used to determine a projection matrix and a view matrix.

10. The method according to claim 8, characterized in that After displaying the rendering data in the virtual scene, the method further includes: Sending the acquired head tracking data, eye tracking data and hand tracking data to the server; Receiving dynamic rendering data sent by the server; The dynamic rendering data is displayed in the virtual scene.

Images