Method for visualizing rendering performance, server and readable storage medium

Abstract

The embodiment of the invention provides a method for visualizing rendering performance, a server and a readable storage medium. The method includes: receiving a plurality of gestures and a plurality of smoothness states associated with the plurality of gestures, respectively, from a first client device; and outputting a rendered performance map based on the plurality of gestures and the associated plurality of smoothness states, wherein the rendered performance map includes a plurality of directional smoothness indicators.

Description

Technical Field

[0001] The present invention generally relates to a mechanism for visualizing performance, and more particularly, to a method, a server, and a computer-readable storage medium for visualizing rendering performance. Background Art

[0002] In the field of virtual reality (VR), the term "remote rendering" refers to the process of offloading the rendering task of a VR experience from a local VR device (such as a VR head-mounted display (HMD) or computer) to a remote server or powerful computing resource. Instead of the VR device handling all the complex graphics calculations required for a high-quality VR experience, these calculations are performed on the remote rendering device and only the resulting display information is sent back to the VR device.

[0003] Remote rendering mechanisms provide a solution that allows low-end VR devices to experience VR content by streaming from a remote powerful graphics server, such as playing SteamVR content on a mobile VR device.

[0004] Generally, the smoothness of remote rendering is affected by two factors: (1) the quality of wireless communication between the VR device and the remote rendering device; and (2) the rendering process (e.g., rendering, decoding, encoding, etc.) performed on the VR device and / or the remote rendering device.

[0005] However, when a user has a choppy VR experience, the user may question the quality of the VR device and / or VR application, but the choppy VR experience may actually be because the user is in a location with poor wireless communication quality and / or a heavily loaded rendering process. Summary of the invention

[0006] Therefore, the present invention relates to a method, a server and a computer-readable storage medium for visualizing rendering performance, which can be used to solve the above technical problems.

[0007] An embodiment of the present invention provides a method for visualizing rendering performance, comprising: receiving, by a server, a plurality of postures and a plurality of smoothness states respectively associated with the plurality of postures from a first client device; and outputting, by the server, a rendering performance graph based on the plurality of postures and the plurality of associated smoothness states, wherein the rendering performance graph includes a plurality of directional smoothness indicators.

[0008] An embodiment of the present invention provides a server, comprising a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and configured to access the program code to execute: receiving a plurality of gestures and a plurality of smoothness states respectively associated with the plurality of gestures from a first client device; and outputting a rendering performance graph based on the plurality of gestures and the plurality of related smoothness states, wherein the rendering performance graph includes a plurality of directional smoothness indicators.

[0009] An embodiment of the present invention provides a non-temporary computer-readable storage medium, which records an executable computer program, and the executable computer program is loaded by a server to perform the following steps: receiving a plurality of postures and a plurality of smoothness states respectively associated with the plurality of postures from a first client device; and outputting a rendering performance graph based on the plurality of postures and the associated plurality of smoothness states, wherein the rendering performance graph includes a plurality of directional smoothness indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0011] Figure 1 A schematic diagram of a server and some client devices according to an embodiment of the present invention is shown.

[0012] Figure 2 A flow chart of a method for visualizing rendering performance according to an embodiment of the present invention is shown.

[0013] Figure 3A and Figure 3B A schematic diagram of a rendering performance graph corresponding to the same reference field according to an embodiment of the present invention is shown.

[0014] Figure 4 Drawing basis Figure 3A and Figure 3B Schematic diagram of the integrated rendering performance graph after adjusting the spatial configuration of the reference field.

[0015] Figure 5 FIG. 4 is a schematic diagram illustrating MR visual content rendered by a first client device according to an embodiment of the present invention.

[0016] Description of Figure Numbers

[0017] 10: Server;

[0018] 11: first client device;

[0019] 12: second client device;

[0020] 31a, 31b: Rendering performance graphs;

[0021] 40: Integrated rendering performance graph;

[0022] 50: MR visual content;

[0023] 51, 52, 53: virtual objects;

[0024] 102: storage circuit;

[0025] 104: processor;

[0026] 310, 320, 330: Directional smoothness indicator;

[0027] 311, 321, 331: smoothness indicators;

[0028] 312, 322, 332: direction indicators;

[0029] 399, 399a: base station;

[0030] 599: Through view;

[0031] B1: other postures;

[0032] B2: other smoothness states;

[0033] P1, Pi, PN: posture;

[0034] S1, Si, SN: smoothness status;

[0035] S210, S220: steps;

[0036] T1, Ti, TN: time point. DETAILED DESCRIPTION

[0037] Reference will now be made in detail to the present preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and description to refer to the same or like parts.

[0038] See also Figure 1 , which depicts a schematic diagram of a server and some client devices according to an embodiment of the present invention. Figure 1 In the embodiment, the server 10 can be connected to the first client device 11 and the second client device 12. In various embodiments, the server 10, the first client device 11 and / or the second client device 12 can be any smart device / computer device. In some embodiments, the server 10 can be a central device that can process / analyze the data / information provided by the first client device 11 and / or the second client device 12.

[0039] In some embodiments, the first client device 11 and / or the second client device 12 may be a host of visual content that provides reality services such as HMD (e.g., VR services, augmented reality (AR) services, mixed reality (MR) services, etc.), but the present invention is not limited to this.

[0040] In an embodiment of the present invention, the operation performed between the server 10 and the first client device 11 may be similar to the operation performed between the server 10 and the second client device 12. In this case, the following discussion will mainly be conducted by introducing the operation performed between the server 10 and the first client device 11, and those skilled in the art should be able to deduce how to perform the operation between the server 10 and the second client device 12 accordingly, but the present invention is not limited thereto.

[0041] In some embodiments, the server 10 may be a remote rendering device of the first client device 11 and / or the second client device 12. In some other embodiments, the remote rendering device of the first client device 11 and / or the second client device 12 may be a device other than the server 10.

[0042] For better understanding, the server 10 will be assumed to be a remote rendering device of the first client device 11 (and / or the second client device 12) in the following discussion, but the present invention is not limited thereto. In this case, the first client device 11 (e.g., HMD) can transmit the posture (e.g., 6 degrees of freedom (DOF) data) of the first client device 11 to the server 10, and the server 10 can render the visual content (e.g., VR content) of the reality service application (e.g., VR application) provided by the first client device 11 accordingly.

[0043] In some embodiments, the first client device 11 may detect its posture by using built-in motion detection circuitry (e.g., an inertia measurement unit (IMU)) and / or by executing a tracking mechanism (e.g., inside-out and / or outside-in), but the present invention is not limited thereto.

[0044] In an embodiment of the present invention, each posture may include a corresponding position and a corresponding orientation, but the present invention is not limited thereto.

[0045] Once the server 10 has finished rendering the visual content, the server 10 may encode the rendered visual content into a data stream and transmit the data stream to the first client device 11. The first client device 11 may then decode the data stream into the visual content and display the visual content for the user to view.

[0046] However, if the quality of wireless communication between the server 10 and the first client device 11 is poor, the gesture of the first client device 11 may not be correctly / effectively transmitted to the server 10. Likewise, the server 10 may not be able to correctly / effectively transmit the data stream corresponding to the rendered visual content to the first client device 11. In either case, the user may have an unsmooth experience when interacting with the reality service application.

[0047] In addition, if the posture of the first client device 11 indicates that the user is facing a relatively complex virtual scene in the virtual environment of the reality service application, not only may the server 10 need more time to render and / or encode the corresponding visual content, but the first client device 11 will also need more time to decode the relevant data stream. In this case, the user may also have an unsmooth experience when interacting with the reality service application.

[0048] As can be seen from the above, a jerky experience may result from the user being in a location / place / area with poor wireless communication quality, and / or the user being in a location / place / area that causes a heavy-load rendering process.

[0049] From another perspective, if users can avoid staying at those locations / places / areas that have poor wireless communication quality and / or result in a heavy-loaded rendering process, the users will have a less jerky experience when interacting with the reality service application.

[0050] Therefore, the embodiments of the present invention provide a solution to the above technical problems, which will be described in detail later.

[0051] exist Figure 1 In the embodiment, the storage circuit 102 is one or a combination of the following: static or mobile random access memory (RAM), read-only memory (ROM), flash memory, hard disk or any other similar device, and the storage circuit records multiple modules and / or program codes that can be executed by the processor 104.

[0052] The processor 104 may be coupled to the storage circuit 102, and the processor 104 may be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc.

[0053] In some embodiments, the first client device 11 and / or the second client device 12 may have a structure similar to that of the server 10 , but the present invention is not limited thereto.

[0054] In an embodiment of the present invention, the processor 104 may access the modules and / or program codes stored in the storage circuit 102 to implement the method for visualizing rendering performance provided in the present invention, which will be further discussed below.

[0055] See also Figure 2 , which depicts a flow chart of a method for visualizing rendering performance according to an embodiment of the present invention. The method of this embodiment can be Figure 1 The server 10 in the embodiment is executed and will be used below Figure 1 Description of the components shown in Figure 2 The details of each step in.

[0056] In step S210 , the processor 104 receives a plurality of gestures and a plurality of smoothness states respectively associated with the plurality of gestures from the first client device 11 .

[0057] In an embodiment of the present invention, the user of the first client device 11 may move around in the reference field. In some embodiments, the reference field may be a game area where the user experiences the reality service application (e.g., VR application). In this case, the user may watch the virtual environment of the reality service application displayed by the first client device 11 while moving around, but the present invention is not limited thereto.

[0058] During the time when the first client device 11 and the user move around in the reference field, the first client device 11 may detect its posture and the related smoothness state at different time points during the time.

[0059] In an embodiment of the present invention, the plurality of gestures may include a first gesture, the plurality of smoothness states may include a first smoothness state associated with the first gesture, and the first smoothness state is detected when the first client device 11 has the first gesture.

[0060] For example, assuming that the period during which the first client device 11 and the user move around in the reference field includes time point T1 to time point TN (N is a positive integer), the first client device 11 can detect each gesture and related smoothness state from time point T1 to time point TN.

[0061] For better understanding, the i-th time point (i is an index from 1 to N) among the time point T1 to the time point TN may be referred to as the time point Ti. In this case, the first client device 11 may detect a gesture and a related smoothness state at the time point Ti, wherein the gesture detected at the time point Ti may be referred to as a gesture Pi, and the smoothness state related to the gesture Pi may be referred to as a smoothness state Si, but the present invention is not limited thereto.

[0062] In this case, the gestures received by the server 10 may include gestures P1 to PN, and the smoothness status received by the server 10 may include smoothness status S1 to SN, but the present invention is not limited thereto.

[0063] In an embodiment of the present invention, the first smoothness status may include a first wireless communication quality of the first client device 11 communicating with a remote rendering device (eg, the server 10 ) of the first client device 11 .

[0064] In different embodiments, the first wireless communication quality of the first smoothness state can be characterized by, for example, a received signal strength indicator (RSSI), a signal-to-noise ratio (SNR), a signal to interference & noise ratio (SINR), data throughput, data loss rate, etc., but the present invention is not limited thereto.

[0065] For example, the smoothness state Si associated with the posture Pi may include the corresponding wireless communication quality, which may be represented by the RSSI determined by the first client device 11 at the time point Ti, but the invention is not limited thereto.

[0066] In some embodiments, the first smoothness state may also include a first rendering state of a rendering process executed on at least one of the first client device and a remote rendering device of the first client device.

[0067] In different embodiments, the first rendering state of the rendering process may include at least one of a rendering frame rate of a remote rendering device (e.g., server 10), a rendering time of the remote rendering device, an encoding time of the remote rendering device, an encoding frame rate of the remote rendering device, a decoding rate of the first client device 11, a decoding time of the first client device 11, and a client rendering rate of the first client device 11.

[0068] For example, the smoothness state Si associated with the posture Pi may also include a corresponding rendering state, which may include one or more of the above-mentioned factors (e.g., rendering frame rate, rendering time, etc.) detected at the time point Ti, but the present invention is not limited to this.

[0069] In an embodiment assuming that the remote rendering device is the server 10, part of the above information may be determined / detected by the server 10 itself, such as rendering frame rate, rendering time, encoding time, encoding frame rate, but the present invention is not limited thereto.

[0070] As mentioned above, in the case where the first client device 11 is displaying the virtual environment of the real service application in the reference field, the postures P1 to PN and the related smoothness states S1 to SN may be detected by the first client device 11. In one embodiment, the correspondence between the virtual environment and the reference field may be fixed.

[0071] That is, if the first client device 11 has the same posture at different time points, the first client device 11 will display the same virtual scene of the virtual environment for the user to watch. From another perspective, if the postures at two time points indicate that the user is at the same position with the same orientation in the reference field at the two time points, the user will watch the same virtual scene at the two time points.

[0072] More specifically, the virtual environment corresponding to the reference field will not change over time, but the present invention is not limited thereto.

[0073] In step S220 , the processor 104 outputs a rendering performance map based on the plurality of postures P1 to postures PN and the associated plurality of smoothness states S1 to smoothness states SN, wherein the rendering performance map includes a plurality of directional smoothness indicators.

[0074] In one embodiment, the plurality of directional smoothness indicators may include a first directional smoothness indicator determined based on a first posture and an associated first smoothness state.

[0075] In one embodiment, the first posture may include a position and an orientation, and the first directional smoothness indicator may include a first smoothness indicator and a first direction indicator. In addition, the first smoothness indicator corresponds to a first smoothness state, the first position of the first smoothness indicator in the rendering performance graph corresponds to the position of the first posture, and the first direction indicator corresponds to the orientation of the first posture.

[0076] In one embodiment, the visual type of the first smoothness indicator is determined based on the first smoothness state, and the direction in which the first direction indicator points is determined based on the orientation of the first gesture.

[0077] For better understanding, Figure 3A and Figure 3B will be used as an example, where Figure 3A and Figure 3B A schematic diagram of a rendering performance graph corresponding to the same reference field according to an embodiment of the present invention is shown.

[0078] In an embodiment of the present invention, each rendering performance graph mentioned above may be specifically referred to as a remote rendering performance graph that depicts the performance of remote rendering, but the present invention is not limited thereto.

[0079] In an embodiment of the present invention, if the smoothness state corresponding to a certain smoothness indicator is determined to be good (e.g., good wireless communication quality and good rendering state), the visual type of this specific smoothness indicator may be drawn as a hollow circle. In addition, if the smoothness state corresponding to the specific smoothness indicator is determined to be acceptable (e.g., acceptable wireless communication quality and acceptable rendering state), the visual type of this specific smoothness indicator may be drawn as a hollow triangle. In addition, if the smoothness state corresponding to the specific smoothness indicator is determined to be poor (e.g., poor wireless communication quality and poor rendering state), the visual type of this specific smoothness indicator may be drawn as a hollow square, but the present invention is not limited thereto.

[0080] Based on the above assumptions, since Figure 3A All smoothness indicators shown in FIG. 1 are shown as hollow circles, so Figure 3A It can be understood that only the directional smoothness indicator corresponding to the good smoothness state is drawn. Figure 3B The smoothness indicators shown in FIG. 1 are shown as hollow triangles or squares, so Figure 3B It can be understood that only the directional smoothness indicator corresponding to the acceptable / poor smoothness state is shown, but the present invention is not limited thereto.

[0081] exist Figure 3AIn FIG. 3 , the rendering performance graph 31 a may correspond to a top view of the reference field and include a plurality of directional smoothness indicators, and each directional smoothness indicator may include a smoothness indicator (shown as a hollow circle) and a direction indicator (shown as an arrow). Figure 3A In FIG. 1 , each directional smoothness indicator is illustrated as a combination of a hollow circle and an arrow, but the present invention is not limited thereto. Figure 3A In the present invention, for the sake of brevity, the direction indicators (eg, arrows) in some of the directional smoothness indicators may be regarded as omitted, but the present invention is not limited thereto.

[0082] Since the rendering performance graph 31 a depicts a plurality of directional smoothness indicators, the directional smoothness indicator 310 will be used as an example for discussion, and the characteristics of other directional smoothness indicators should be deducible by those of ordinary skill in the art.

[0083] exist Figure 3A , the directional smoothness indicator shown may correspond to portions of postures P1 to PN (and associated smoothness states S1 to SN), respectively. For better understanding, the directional smoothness indicator 310 may be assumed to correspond to posture P1 and associated smoothness state S1.

[0084] In this case, the directional smoothness indicator 310 may be determined based on the posture P1 and the related smoothness state S1. Specifically, the directional smoothness indicator 310 may include a smoothness indicator 311 and a direction indicator 312, wherein the position of the smoothness indicator 311 in the rendering performance graph 31a corresponds to the position of the posture P1, and the direction indicator 312 corresponds to the orientation of the posture P1.

[0085] That is, the position of the smoothness indicator 311 in the rendering performance graph 31 a is the position of the posture P1 , and the direction pointed by the direction indicator 312 is determined based on the orientation of the posture P1 .

[0086] In one embodiment, the smoothness indicator 311 corresponds to the smoothness state S1. In one embodiment, the visual type of the smoothness indicator 311 is determined based on the smoothness state S1. Figure 3A In FIG. 3 , it is assumed that the smoothness state S1 is good, which causes the smoothness indicator 311 to be shown as a visual type of a hollow circle, but the present invention is not limited thereto.

[0087] Therefore, based on the directional smoothness indicator 310 , it can be observed that as long as the client device is located at the position indicated by the smoothness indicator 311 and faces the direction pointed by the direction indicator 312 , a good smoothness state can be achieved.

[0088] exist Figure 3BIn FIG. 3 , the rendering performance graph 31 b may also correspond to a top view of the reference field and include a plurality of directional smoothness indicators, and each directional smoothness indicator may include a smoothness indicator (shown as a hollow triangle / square) and a direction indicator (shown as an arrow). Figure 3B In the figure, each directional smoothness indicator is shown as a combination of a hollow triangle and an arrow or a combination of a hollow square and an arrow, but the present invention is not limited thereto. Figure 3B In the present invention, for the sake of brevity, the direction indicators (eg, arrows) in some of the directional smoothness indicators may be regarded as omitted, but the present invention is not limited thereto.

[0089] Since the rendering performance graph 31 b depicts numerous directional smoothness indicators, directional smoothness indicator 320 and directional smoothness indicator 330 will be used as examples for discussion, and the characteristics of other directional smoothness indicators should be deducible by those of ordinary skill in the art.

[0090] exist Figure 3B , the directional smoothness indicator shown may correspond to another portion of postures P1 to PN (and associated smoothness states S1 to SN), respectively. For better understanding, directional smoothness indicator 320 may be assumed to correspond to posture P2 and associated smoothness state S2.

[0091] In this case, the directional smoothness indicator 320 may be determined based on the posture P2 and the related smoothness state S2. Specifically, the directional smoothness indicator 320 may include a smoothness indicator 321 and a direction indicator 322, wherein the position of the smoothness indicator 321 in the rendering performance graph 31b corresponds to the position of the posture P2, and the direction indicator 322 corresponds to the orientation of the posture P2.

[0092] That is, the position of the smoothness indicator 321 in the rendering performance graph 31 b is the position of the posture P2 , and the direction pointed by the direction indicator 322 is determined based on the orientation of the posture P2 .

[0093] In one embodiment, the smoothness indicator 321 corresponds to the smoothness state S2. In one embodiment, the visual type of the smoothness indicator 321 is determined based on the smoothness state S2. Figure 3B In FIG. 3 , it is assumed that the smoothness state S2 is acceptable, which causes the smoothness indicator 321 to be shown as a visual type of a hollow triangle, but the present invention is not limited thereto.

[0094] Therefore, based on the directional smoothness indicator 320 , it can be observed that as long as the client device is located at the position indicated by the smoothness indicator 321 and faces the direction pointed by the direction indicator 322 , an acceptable smoothness state can be achieved.

[0095] Additionally, directional smoothness indicator 330 may be assumed to correspond to posture PN and associated smoothness state SN.

[0096] In this case, the directional smoothness indicator 330 may be determined based on the posture PN and the associated smoothness state SN. Specifically, the directional smoothness indicator 330 may include a smoothness indicator 331 and a direction indicator 332, wherein the position of the smoothness indicator 331 in the rendering performance graph 31b corresponds to the position of the posture PN, and the direction indicator 332 corresponds to the orientation of the posture PN.

[0097] That is, the position of the smoothness indicator 331 in the rendering performance graph 31 b is the position of the posture PN, and the direction pointed by the direction indicator 332 is determined based on the orientation of the posture PN.

[0098] In one embodiment, the smoothness indicator 331 corresponds to the smoothness state SN. In one embodiment, the visual type of the smoothness indicator 331 is determined based on the smoothness state SN. Figure 3B In FIG. 3 , it is assumed that the smoothness status SN is bad, which causes the smoothness indicator 331 to be shown as a visual type of a hollow square, but the present invention is not limited thereto.

[0099] Therefore, based on the directional smoothness indicator 330 , it can be observed that as long as the client device is located at the position indicated by the smoothness indicator 331 and facing the direction pointed by the direction indicator 332 , the poor smoothness state is achieved.

[0100] In one embodiment, Figure 3A and Figure 3B can be combined into another rendering performance graph (called an integrated rendering performance graph), which can depict Figure 3A and Figure 3B That is, the integrated rendering performance graph will plot Figure 3A and Figure 3B All directional smoothness indicators in the embodiment of the present invention are provided, but the present invention is not limited thereto.

[0101] In one embodiment, the integrated rendering performance graph may be designed with a filtering function that allows the user to specify viewing of directional smoothness indicators corresponding to good / acceptable / poor smoothness states. For example, if the user wants to view only directional smoothness indicators corresponding to a good smoothness state, the user may activate the filtering function corresponding to a good smoothness state so that the user can view only directional smoothness indicators corresponding to a good smoothness state. Figure 3AFor another example, if the user wants to view only the directional smoothness indicators corresponding to the acceptable and poor smoothness states, the user can activate the filtering function corresponding to the acceptable and poor smoothness states so that the user can view Figure 3B The rendering performance diagram 31b is shown in FIG. 31 , but the present invention is not limited thereto.

[0102] From another perspective, the rendering performance graph mentioned above can be understood as providing information about which areas of the reference field allow the user to have a smooth experience (for example, Figure 3A ) and which areas of the reference field may cause an unsmooth experience (e.g., Figure 3B information on the area indicated by the directional smoothness indicator in the image.

[0103] In this case, the processor 104 may display the rendering performance graph 31a, the rendering performance graph 31b and / or the integrated rendering performance graph so that personnel (e.g., developers / managers / staff) who maintain the real service application can prevent users from having an unsmooth experience by, for example, adjusting the spatial configuration of the reference field (e.g., removing obstacles, enhancing signal strength, adding base stations 399, etc.) and / or simplifying the virtual scene in the virtual environment.

[0104] See also Figure 4 , which is drawn based on Figure 3A and Figure 3B Schematic diagram of the integrated rendering performance graph after adjusting the spatial configuration of the reference field. Figure 4 In the foregoing, for the sake of brevity, the direction indicators in some of the directional smoothness indicators may be regarded as omitted, but the present invention is not limited thereto.

[0105] exist Figure 4 In the figure, it is assumed that the spatial configuration of the reference field has been adjusted by placing another base station 399a in the reference field, but the present invention is not limited to this.

[0106] from Figure 4 It can be seen that although the integrated rendering performance map 40 is controlled to depict directional smoothness indicators corresponding to good, acceptable, and poor smoothness states, Figure 4 All directional smoothness indicators in the sleekness state appear to correspond to a good smoothness state. That is, the user is in the state corresponding to Figure 4 When interacting with real-world service applications in your reference domain, you will have a less frictionless experience.

[0107] In addition, although only the operations between the server 10 and the first client device 11 are described in the above embodiment, similar operations may also be performed between the server 10 and the second client device 12 (and / or even other client devices). For example, the processor 104 may receive a plurality of other gestures B1 and a plurality of other smoothness states B2 respectively related to the plurality of other gestures B1 from the second client device 12, and the processor 104 may modify the rendering performance graph based on the plurality of other gestures B1 and the plurality of other other smoothness states B2 related thereto.

[0108] For example, processor 104 may modify rendering performance graph 31a and / or rendering performance graph 31b by adding other directional smoothness indicators determined based on other gestures B1 and other smoothness states B2 from second client device 12, but the present invention is not limited thereto.

[0109] In some embodiments, the user may also be notified to avoid specific areas that cause a non-smooth experience in the reference field. For example, the processor 104 may send one or more of the above rendering performance graphs to the first client device 11 .

[0110] In one embodiment, the first client device 11 may directly display the received rendering performance graph for the user to view, so that the user may adjust the position of the user in the reference field accordingly.

[0111] In one embodiment, after receiving the rendering performance graph from the server 10, the first client device 11 may render mixed reality (MR) visual content based on a field of view (FOV) of the first client device 11 and a plurality of directional smoothness indicators in the rendering performance graph. In one embodiment, the mixed reality (MR) visual content includes a plurality of virtual objects rendered based on the plurality of directional smoothness indicators.

[0112] In one embodiment, the first client device 11 (e.g., HMD) can use its front camera with FOV to capture an image of the real world scene in front of the first client device 11, and render a pass-through view accordingly as the background of the MR visual content. In this case, the user can see the real world scene in front of the user in the MR visual content.

[0113] See also Figure 5 , which is a schematic diagram of MR visual content rendered by a first client device according to an embodiment of the present invention. Figure 5In the embodiment, the MR visual content 50 may include a through-view 599 and virtual objects 51 to 53, wherein the through-view 599 may correspond to a real-world scene in front of the user, and the virtual objects 51 to 53 may be rendered based on the directional smoothness indicator in the rendering performance map received by the first client device 11.

[0114] In an embodiment, for a directional smoothness indicator corresponding to a good smoothness state, the first client device 11 may render, for example, a light beam having a first color (eg, green) at a corresponding position in the reference field indicated by the directional smoothness indicator.

[0115] exist Figure 5 In the example, it is assumed that the virtual object 51 (e.g., a light beam having a first color) is rendered based on a directional smoothness indicator corresponding to a good smoothness state. In this case, the user will know that the area indicated by the virtual object 51 can allow the user to have a smooth experience.

[0116] In an embodiment, for a directional smoothness indicator corresponding to an acceptable smoothness state, first client device 11 may render, for example, a light beam having a second color (eg, blue) at a corresponding position in the reference field indicated by the directional smoothness indicator.

[0117] exist Figure 5 In the example, it is assumed that the virtual object 52 (e.g., a light beam having a second color) is rendered based on a directional smoothness indicator corresponding to an acceptable smoothness state. In this case, the user will know that the area indicated by the virtual object 52 may result in a less smooth experience.

[0118] In an embodiment, for a directional smoothness indicator corresponding to a bad smoothness state, first client device 11 may render, for example, a light beam having a third color (eg, red) at a corresponding position in the reference field indicated by the directional smoothness indicator.

[0119] exist Figure 5 In the example, it is assumed that the virtual object 53 (e.g., a light beam having a third color) is rendered based on a directional smoothness indicator corresponding to a poor smoothness state. In this case, the user will know that the area indicated by the virtual object 53 should be avoided to avoid an unsmooth experience.

[0120] In some embodiments, the MR visual content 50 may also be rendered by the server 10 and provided to the first client device 11 for display, but the present invention is not limited thereto.

[0121] The present invention also provides a computer-readable storage medium for executing the method for visualizing rendering performance. The computer-readable storage medium is composed of a plurality of program instructions (e.g., setting program instructions and deployment program instructions) embodied herein. These program instructions can be loaded into and executed by the server 10 to perform the method for visualizing rendering performance and the functions of the server 10 described above.

[0122] In summary, the embodiments of the present invention provide a technical solution for integrating gestures and related smoothness states reported by a first client device into a rendering performance graph with a directional smoothness indicator, wherein the directional smoothness indicator visualizes the gestures and related smoothness states. In this case, the rendering performance graph can show which areas of the reference field allow remote rendering to be performed efficiently, and which areas of the reference field may reduce the efficiency of remote rendering.

[0123] Therefore, the rendering performance map can be used as a reference to improve the spatial configuration of the reference field, which can improve the user experience.

[0124] In addition, the rendering performance map may be used to render the corresponding MR visual content, so that the directional smoothness indicators therein may be visualized as some specific areas in the real-world scene for the user to view.

[0125] It will be apparent to those skilled in the art that various modifications and variations may be made to the structure of the present invention without departing from the scope or spirit of the present invention. In view of the foregoing, it is desired that the present invention encompasses modifications and variations of the present invention, provided that the modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. A method for visualizing rendering performance, characterized in that include: receiving, by the server from the first client device, a plurality of gestures and a plurality of smoothness states respectively associated with the plurality of gestures; as well as A rendering performance map is outputted by the server based on the plurality of gestures and the associated plurality of smoothness states, wherein the rendering performance map includes a plurality of directional smoothness indicators.

2. The method for visualizing rendering performance of claim 1, wherein the plurality of gestures includes a first gesture, the plurality of smoothness states includes a first smoothness state associated with the first gesture, and the first smoothness state is detected when the first client device has the first gesture. 3 . The method for visualizing rendering performance according to claim 2 , wherein the first smoothness status comprises a first wireless communication quality of the first client device in communication with a remote rendering device of the first client device. The method for visualizing rendering performance of claim 3 , wherein the server is the remote rendering device of the first client device. 5 . The method for visualizing rendering performance of claim 3 , wherein the first smoothness state further comprises a first rendering state of a rendering process executed on at least one of the first client device and the remote rendering device of the first client device.

6. The method for visualizing rendering performance according to claim 5, wherein the first rendering state of the rendering process comprises at least one of a rendering frame rate of the remote rendering device, a rendering time of the remote rendering device, an encoding time of the remote rendering device, an encoding frame rate of the remote rendering device, a decoding rate of the first client device, a decoding time of the first client device, and a client rendering rate of the first client device. 7 . The method for visualizing rendering performance of claim 1 , wherein the plurality of gestures and the associated plurality of smoothness states are detected by the first client device in a reference field.

8. A method for visualizing rendering performance according to claim 7, wherein the plurality of postures and the associated plurality of smoothness states are detected by the first client device when the first client device is displaying a virtual environment of a real service application in the reference field, and the correspondence between the virtual environment and the reference field is fixed. 9 . The method for visualizing rendering performance of claim 2 , wherein the plurality of directional smoothness indicators comprises a first directional smoothness indicator determined based on the first gesture and the associated first smoothness state.

10. The method for visualizing rendering performance of claim 9, wherein the first posture comprises a position and an orientation, and the first directional smoothness indicator comprises a first smoothness indicator and a first direction indicator; The first smoothness indicator corresponds to the first smoothness state, a first position of the first smoothness indicator in the rendering performance graph corresponds to the position of the first posture, and the first direction indicator corresponds to the orientation of the first posture. 11 . The method for visualizing rendering performance of claim 10 , wherein a visual type of the first smoothness indicator is determined based on the first smoothness state, and a direction pointed by the first direction indicator is determined based on the orientation of the first gesture.

12. The method for visualizing rendering performance according to claim 1, further comprising: receiving, by the server from a second client device, a plurality of other gestures and a plurality of other smoothness states respectively associated with the plurality of other gestures; as well as The rendering performance graph is modified by the server based on the plurality of other gestures and the associated plurality of other smoothness states.

13. The method for visualizing rendering performance according to claim 1, further comprising: The rendering performance map is transmitted by the server to the first client device.

14. The method for visualizing rendering performance according to claim 13, further comprising: Mixed reality visual content is rendered by the first client device based on the field of view of the first client device and the plurality of directional smoothness indicators in the rendering performance map, wherein the mixed reality visual content includes a plurality of virtual objects rendered based on the plurality of directional smoothness indicators.

15. The method for visualizing rendering performance according to claim 1, wherein outputting the rendering performance graph based on the plurality of gestures and the associated plurality of smoothness states comprises: The rendering performance graph is displayed by the server.

16. A server, characterized in that: include: A storage circuit for storing program codes; as well as a processor coupled to the memory circuit and configured to access the program code to execute: receiving, from a first client device, a plurality of gestures and a plurality of smoothness states respectively associated with the plurality of gestures; as well as A rendering performance map is output based on the plurality of gestures and the associated plurality of smoothness states, wherein the rendering performance map includes a plurality of directional smoothness indicators.

17. The server of claim 16, wherein the plurality of gestures includes a first gesture, the plurality of smoothness states includes a first smoothness state associated with the first gesture, and the first smoothness state is detected when the first client device has the first gesture; wherein the first smoothness state comprises a first wireless communication quality of the first client device communicating with a remote rendering device of the first client device; The first smoothness state further includes a first rendering state of a rendering process performed on at least one of the first client device and the remote rendering device of the first client device.

18. The server of claim 16, wherein the plurality of gestures and the associated plurality of smoothness states are detected by the first client device in a reference field; The plurality of postures and the associated plurality of smoothness states are detected by the first client device when the first client device is displaying a virtual environment of a real service application in the reference field, and the correspondence between the virtual environment and the reference field is fixed.

19. The server of claim 16, wherein the plurality of gestures includes a first gesture, the plurality of smoothness states includes a first smoothness state associated with the first gesture, and the first smoothness state is detected when the first client device has the first gesture; The plurality of directional smoothness indicators include a first directional smoothness indicator determined based on the first posture and the associated first smoothness state.

20. A computer-readable storage medium recording an executable computer program, wherein the executable computer program is loaded by a server to perform the following steps: receiving, from a first client device, a plurality of gestures and a plurality of smoothness states respectively associated with the plurality of gestures; and A rendering performance map is output based on the plurality of gestures and the associated plurality of smoothness states, wherein the rendering performance map includes a plurality of directional smoothness indicators.

Images