End-to-end delay measurement method and device, storage medium and computer program product

Abstract

The invention provides an end-to-end delay measurement and calculation method and device, a storage medium and a computer program product, and is applied to a cloud end, and the method comprises the steps: responding to an obtained delay measurement and calculation instruction started by a client, starting a delay measurement and calculation auxiliary program, and placing the delay measurement and calculation auxiliary program on a cloud rendering application; obtaining a delay measurement and calculation request sent by the client, and forwarding the delay measurement and calculation request to the delay measurement and calculation auxiliary program; the delay measurement and calculation request is generated by the client in response to a touch operation for a first delay measurement and calculation button in the delay measurement and calculation interface; and obtaining a delay information picture and a cloud rendering picture displayed by the delay measurement auxiliary program, and transmitting the pictures back to the client for display. And the end-to-end delay can be accurately measured and calculated.

Description

Technical Field

[0001] This application relates to the field of transmission and bearer technology, and in particular to an end-to-end delay measurement method, device, storage medium and computer program product. Background Technology

[0002] The end-to-end process of real-time cloud rendering typically involves the cloud service sending video frames rendered by the cloud application to the client for display. The client then sends control commands to the cloud for response. Upon receiving the control commands, the cloud needs to translate the control command events into keyboard and mouse events and add them to the operating system's input event queue for processing one by one. However, the translation efficiency of control commands, the response of the cloud application to the operation, and the performance of the server and the current load of the environment are all related. This leads to the neglect of the latency caused by the translation of control commands and the response of the cloud application, resulting in a large error in the final calculated end-to-end latency data. Summary of the Invention

[0003] This application provides an end-to-end latency measurement method, device, storage medium, and computer program product that can accurately measure end-to-end latency.

[0004] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0005] Firstly, this application proposes an end-to-end latency measurement method for use in the cloud; the method includes:

[0006] In response to the latency calculation command initiated by the client, the latency calculation auxiliary program is launched and placed on top of the cloud rendering application;

[0007] The delay calculation request sent by the client is obtained, and the delay calculation request is forwarded to the delay calculation auxiliary program; the delay calculation request is generated by the client in response to the touch operation of the first delay calculation button in the delay calculation interface;

[0008] The delay information screen and cloud rendering screen displayed by the delay calculation auxiliary program are obtained and sent back to the client for display.

[0009] Secondly, this application proposes an end-to-end latency measurement method, applied to a client side; the method includes:

[0010] In response to the start delay calculation command, a delay calculation interface is displayed; the delay calculation interface includes a first delay calculation button and a first display area;

[0011] In response to a touch operation on the first delay calculation button in the delay calculation interface, a delay calculation request is generated and sent to the cloud;

[0012] In response to the latency information screen and cloud rendering screen transmitted from the cloud, the latency information screen is displayed in the first display area, and the cloud rendering screen is used as the interface background of the latency calculation interface.

[0013] Thirdly, this application proposes an electronic device, comprising: a processor and a memory for storing a computer program capable of running on the processor.

[0014] When the processor runs the computer program, it executes the steps of any of the methods described above on the cloud side, or executes the steps of any of the methods described above on the client side.

[0015] Fourthly, this application proposes a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above on the cloud side, or implements the steps of any of the methods described above on the client side.

[0016] Fifthly, this application proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described above on the cloud side, or implements the steps of any of the methods described above on the client side.

[0017] This application proposes an end-to-end latency measurement method, device, storage medium, and computer program product applied in the cloud. The method includes: responding to a latency measurement instruction initiated by a client, starting a latency measurement auxiliary program and placing it on top of the cloud rendering application; obtaining a latency measurement request sent by the client and forwarding the latency measurement request to the latency measurement auxiliary program; the latency measurement request is generated by the client in response to a touch operation on the first latency measurement button in the latency measurement interface; obtaining the latency information screen and the cloud rendering screen displayed by the latency measurement auxiliary program and sending them back to the client for display. The above solution introduces a latency measurement auxiliary program. This program receives latency measurement requests and displays latency information, taking into account the time required to receive and respond to these requests. It then sends the latency information and cloud-rendered image back to the client for display. The client can then determine the end-to-end latency based on the latency information displayed on the latency information screen. This approach fully considers the latency caused by receiving and responding to latency measurement requests, enabling accurate measurement of end-to-end latency. Attached Figure Description

[0018] Figure 1 This is an exemplary end-to-end process diagram of real-time cloud rendering provided for an embodiment of this application;

[0019] Figure 2 A flowchart illustrating an end-to-end delay measurement method provided in an embodiment of this application;

[0020] Figure 3 A flowchart illustrating another end-to-end delay measurement method provided in an embodiment of this application;

[0021] Figure 4 This is an exemplary end-to-end latency flowchart provided for an embodiment of this application;

[0022] Figure 5 A schematic diagram illustrating another exemplary end-to-end latency provided for an embodiment of this application;

[0023] Figure 6 A schematic diagram of the interface of an exemplary delay calculation auxiliary program provided for an embodiment of this application;

[0024] Figure 7 A schematic diagram illustrating an exemplary delay calculation auxiliary program display method provided in an embodiment of this application;

[0025] Figure 8 A schematic diagram illustrating an exemplary delay calculation request time and delay calculation request identifier displayed in a delay calculation auxiliary program, provided for embodiments of this application;

[0026] Figure 9 This is a schematic diagram illustrating an exemplary process by which a cloud rendering server sends a response screen with request information back to the client, as provided in an embodiment of this application.

[0027] Figure 10 A schematic diagram illustrating the extraction of delay measurement request time as an example provided in an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the structure of an end-to-end delay measurement device provided in an embodiment of this application;

[0029] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0030] Figure 13 A schematic diagram of another end-to-end delay measurement device provided in an embodiment of this application;

[0031] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0034] In the following description, references to "some embodiments" are made, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," etc., may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Real-time cloud rendering is a technology that migrates content such as applications, desktop environments, and games from the user's local computer to servers in the cloud for processing and rendering. In this way, the visuals seen by the end user and their interactions are rendered in real-time via the cloud. Streaming technology is a key method for achieving cloud rendering; it captures images from the application in real-time, converts them into video streams, and then transmits them over the network to the user's terminal for display. Simultaneously, users can interact with the real-time displayed images using a keyboard, mouse, touchscreen, etc., and the corresponding control commands are captured and sent to the cloud in real-time, enabling real-time control of the application.

[0036] The end-to-end latency of real-time cloud rendering is mainly composed of multiple components. Figure 1 This application provides an exemplary end-to-end process diagram for real-time cloud rendering; as shown in the embodiments of this application. Figure 1 As shown, the end-to-end process of real-time cloud rendering is as follows (the process of the client initiating a cloud rendering request to the cloud service and the cloud service launching the cloud application upon receiving the request is part of the request process and not part of the latency calculation process):

[0037] 1. Application data collection.

[0038] It should be noted that the application capture process involves the streaming service capturing video frames from the current screen or after the cloud application has rendered.

[0039] 2. Hardware coding.

[0040] It should be noted that the hardware encoding process involves the streaming service compressing and encoding the captured screen or video frames to reduce the amount of data for transmission.

[0041] 3. Smooth delivery.

[0042] It should be noted that the smooth transmission process involves smoothing the encoded data into a media stream and then sending it to the client.

[0043] 4. Receive frame.

[0044] It should be noted that the process of receiving and framing involves the client receiving the encoded data and assembling it into a complete frame.

[0045] 5. Hardware decoding.

[0046] It should be noted that the hardware decoding process involves the client decoding the received video frames and restoring them into image frames.

[0047] 6. Hardware rendering.

[0048] It should be noted that the hardware rendering process involves the client rendering the decoded image frames onto the display screen and presenting them to the user.

[0049] 7. Keyboard and mouse capture.

[0050] It should be noted that the keyboard and mouse capture process allows the user to view the remote cloud application screen and perform keyboard and mouse operations as if they were locally accessing it. Simultaneously, the client captures the user's keyboard and mouse input.

[0051] 8. Keyboard and mouse operation commands.

[0052] It should be noted that keyboard and mouse operation commands are sent from the client to the streaming service. The streaming service receives the user's input commands and translates them into machine commands to drive the cloud application to respond. This process is repeated to ensure real-time response to user actions and smooth screen display. Due to the long and complex chain, calculating end-to-end latency presents a significant challenge. In particular, the command translation process essentially involves translating the command events received by the streaming service into keyboard and mouse events and adding them to the operating system's input event queue. The operating system then processes these events one by one according to the order in the queue. During processing, the operating system simulates user input behavior and passes the input events to the active window or the currently focused control. The overall efficiency of command translation is closely related to server performance, the current environmental load, and other factors.

[0053] 9. Keyboard and mouse replay.

[0054] It should be noted that keyboard and mouse playback generates keyboard and mouse operations.

[0055] The above solutions often overlook the latency caused by the translation of control commands and the application's response to control commands, resulting in significant errors in the final calculated end-to-end latency data. In real-time cloud rendering scenarios, the overall end-to-end latency is within 120 milliseconds (ms), while actual measurements show that the time consumed by the translation of control commands and the application's response to control commands fluctuates greatly (5ms-100+ms). As we know from the previously mentioned information, this is mainly closely related to factors such as server performance and the current load of the environment. Ignoring this latency will inevitably introduce a large measurement error. Adding latency measurement auxiliary frames or auxiliary information to the return stream sent by the streaming service increases the performance overhead of the streaming service and wastes downlink bandwidth.

[0056] Based on this, embodiments of this application provide an end-to-end delay measurement method. Figure 2 This is a flowchart illustrating an end-to-end delay measurement method provided in an embodiment of this application; as shown below. Figure 2 As shown, when applied in the cloud, the method includes:

[0057] S201. In response to the latency calculation instruction initiated by the client, start the latency calculation auxiliary program and place it on top of the cloud rendering application.

[0058] It should be noted that "cloud" can be understood as an example of cloud rendering, and the specific scenario can be determined according to the actual situation, without limitation here. As an example, the cloud can be exemplified by cloud gaming, cloud computers, cloud internet cafes, etc. In practical applications, the cloud can be referred to as a cloud rendering server. "Client" can be understood as a terminal device, and the specific term can be determined according to the actual situation, without limitation here. As an example, terminal devices can be mobile phones, computers, tablets, etc.

[0059] It should be noted that the cloud includes streaming services, latency measurement assistant programs, and cloud applications. Cloud applications can be any application; as an example, a cloud application could be a cloud game. Latency measurement commands can be understood as requests sent by the client to the cloud for the corresponding cloud rendering application. The interface of the latency measurement assistant program in the cloud includes a second latency measurement button and a second display area.

[0060] The client-initiated latency calculation instruction can be understood as the instruction generated by the user's touch operation on the third latency calculation button on the client's interface. In response to the received client-initiated latency calculation instruction, the latency calculation auxiliary program is launched and placed on top of the cloud rendering application. For example, the user touches the third latency calculation button on the client, generating a latency calculation instruction which is sent to the cloud rendering server. The cloud rendering server's streaming service, upon receiving the received client-initiated latency calculation instruction, launches the latency calculation auxiliary program and places it on top of the cloud rendering application.

[0061] It should be noted that after the latency calculation assistance program is launched and placed on top of the cloud rendering application, the cloud will send the blank information screen displayed by the latency calculation assistance program and the cloud rendering screen back to the client for display. The blank information screen corresponds to the screen in the first display area. The cloud will also send the screen of the second latency calculation button displayed by the latency calculation assistance program back to the client along with the blank information screen and the cloud rendering screen. The client will then use the received second latency calculation button screen as the screen corresponding to the first latency calculation button in the latency calculation interface.

[0062] S202. Obtain the delay calculation request sent by the client and forward the delay calculation request to the delay calculation auxiliary program; the delay calculation request is generated by the client in response to the touch operation of the first delay calculation button in the delay calculation interface.

[0063] It should be noted that the latency calculation request includes button touch commands, and also includes at least one of the following: latency calculation request time and latency calculation request identifier. The latency calculation request sent by the client is obtained and forwarded to the latency calculation auxiliary program. For example, the user performs a touch operation on the area corresponding to the first latency calculation button on the client, generating a latency calculation request. The client sends the latency calculation request to the streaming service of the cloud rendering server, and the streaming service forwards the latency calculation request to the latency calculation auxiliary program.

[0064] In this embodiment of the application, the delay calculation request includes a button touch instruction; after forwarding the delay calculation request to the delay calculation auxiliary program, it further includes: translating the button touch instruction and triggering the delay calculation auxiliary program to display the delay information screen.

[0065] It should be noted that button touch commands can be understood as the touch commands corresponding to the user's touch operation on the first delay calculation button; in practical applications, button touch commands can be understood as button click event commands. Translating button touch commands can be understood as the cloud rendering server translating the button touch commands into button click actions on the cloud rendering server. After the delay calculation auxiliary program captures the button click action, it clicks the second delay calculation button and triggers the delay calculation auxiliary program to display the delay information screen in the second display area.

[0066] In this embodiment of the application, the delay calculation request further includes at least one of the following: delay calculation request time and delay calculation request identifier; the process of triggering the delay calculation auxiliary program to display the delay information screen specifically includes: triggering the delay calculation auxiliary program, displaying the delay calculation auxiliary program, and displaying the delay calculation request time and / or delay calculation request identifier as the delay information screen.

[0067] It should be noted that the latency measurement request time can be understood as the time when the client sends the latency measurement request to the cloud, which is the client's local time. The latency measurement request identifier is a unique identifier corresponding to the sending of the latency measurement request. In practical applications, the latency measurement request identifier can be recorded as the latency measurement request identity identifier (id). The latency measurement request ID adopts an auto-incrementing ID mechanism, which can be understood as the latency measurement request ID being incremented by the value of the previously sent latency measurement request ID with each subsequent sending. The increment value can be determined based on the actual situation and is not limited here.

[0068] It should be noted that triggering the delay calculation auxiliary program and displaying the delay calculation auxiliary program will display the delay calculation request time and / or delay calculation request identifier as delay information on the screen. For example, after the button click action is applied to the second delay calculation button and the delay calculation auxiliary program is triggered, the delay calculation request time and / or delay calculation request identifier will be displayed as delay information on the second display area.

[0069] In the embodiment of this application, the delay calculation auxiliary program displays the delay calculation request identifier and / or delay calculation request time in the second display area only after capturing the button click action, so that the translation of the control command and the operation response time can be calculated.

[0070] S203. Obtain the delay information screen and cloud rendering screen displayed by the delay calculation auxiliary program and send them back to the client for display.

[0071] It should be noted that the latency information screen includes the latency calculation request time and / or latency calculation request identifier. The process of obtaining the latency information screen and cloud rendering screen displayed by the latency calculation auxiliary program and transmitting them back to the client for display can be illustrated as follows: the cloud rendering server obtains the latency information screen and cloud rendering screen displayed by the latency calculation auxiliary program, including the latency calculation request time and / or latency calculation request identifier, and transmits them back to the client for display.

[0072] It should be noted that the screen displayed by the latency calculation auxiliary program is overlaid on the cloud rendering screen. The screen displayed by the latency calculation auxiliary program includes the screen of the second latency calculation button and the latency information screen.

[0073] The solution in this application embodiment introduces a latency calculation auxiliary program. The streaming service of the cloud rendering server forwards the latency calculation request sent by the client to the latency calculation auxiliary program. This time consumption is almost negligible, reducing the test error introduced by the instruction transmission. The latency calculation request is displayed through the latency calculation auxiliary program, and the displayed latency screen and the cloud rendering screen are sent back to the client for display, so that the client can decode the latency screen and the cloud rendering screen to obtain the accurate end-to-end latency time.

[0074] The solution in this application embeds a latency measurement auxiliary program into the application and utilizes a streaming service to collect data from the application, embedding the latency measurement request time into the transmitted frame. Compared to traditional measurement schemes (which add encoded auxiliary frames or auxiliary information), this application reduces the performance overhead of the streaming service. It also avoids the potential latency increase caused by introducing processing logic.

[0075] This application also provides an end-to-end delay measurement method. Figure 3 A flowchart illustrating another end-to-end delay measurement method provided in this application embodiment is shown below. Figure 3 As shown, applied to the client, the method includes:

[0076] S301. In response to the start delay calculation command, display the delay calculation interface; the delay calculation interface includes a first delay calculation button and a first display area.

[0077] It should be noted that "client" can be understood as a terminal device, and the specific meaning can be determined according to the actual situation; no limitation is made here. As an example, a terminal device can be a mobile phone, computer, tablet, etc.

[0078] It should be noted that the latency calculation command can be understood as a request sent by the client to the cloud for the corresponding command of the cloud rendering application. The latency calculation interface can be understood as the interface returned by the latency calculation auxiliary program in the cloud after receiving the latency calculation command, where the first display area is a blank area. In response to the initiated latency calculation command, the latency calculation interface is displayed. For example, the user touches the third latency calculation button on the client, generating a latency calculation command and sending it to the cloud rendering server. The streaming service of the cloud rendering server responds to the received latency calculation command from the client, starts the latency calculation auxiliary program, and places it on top of the cloud rendering application. The cloud then returns the blank information screen displayed by the latency calculation auxiliary program and the cloud rendering screen to the client for display. The latency calculation interface is the screen returned by the latency calculation auxiliary program, including the second latency calculation button and the first display area, which is a blank information screen. The cloud-based latency calculation auxiliary program includes a second latency calculation button and a second display area; the client's first latency calculation button corresponds to the cloud-based second latency calculation button; and the client's first display area corresponds to the cloud-based second display area.

[0079] S302, In response to a touch operation on the first delay calculation button in the delay calculation interface, generate a delay calculation request and send it to the cloud.

[0080] It should be noted that "cloud" can be understood as an example of cloud rendering; the specific scenario can be determined based on the actual situation and is not limited here. As an example, "cloud" can refer to scenarios such as cloud gaming, cloud computing, and cloud internet cafes. In practical applications, "cloud" can be called a cloud rendering server. "Cloud" includes streaming services, latency measurement auxiliary programs, and cloud applications. Cloud applications can be any application; as an example, a cloud application could be cloud gaming.

[0081] It should be noted that the latency calculation request includes button touch commands, and also includes at least one of the following: latency calculation request time and latency calculation request identifier. In response to a touch operation on the first latency calculation button in the latency calculation interface, a latency calculation request is generated and sent to the cloud. For example, a user performs a touch operation on the area corresponding to the first latency calculation button on the client, generating a latency calculation request. The client then sends the latency calculation request to the streaming service of the cloud rendering server, and the streaming service forwards the latency calculation request to the latency calculation auxiliary program.

[0082] S303. Responding to the latency information screen and cloud rendering screen transmitted from the cloud, the latency information screen is displayed in the first display area, and the cloud rendering screen is used as the interface background of the latency calculation interface.

[0083] It should be noted that the screen displayed by the latency calculation auxiliary program is overlaid on the cloud rendering screen. The screen displayed by the latency calculation auxiliary program includes the screen of the second latency calculation button and the latency information screen.

[0084] In this embodiment of the application, the process of displaying a delay information screen in the first display area specifically includes: displaying a delay information screen including a delay calculation request and / or a delay calculation request identifier in the first display area.

[0085] It should be noted that the latency information screen includes the latency calculation request time and / or latency calculation request identifier. In response to the latency information screen and cloud rendering screen transmitted from the cloud, the latency information screen is displayed in the first display area, and the cloud rendering screen is used as the background of the latency calculation interface. For example, the client responds to the latency information screen and cloud rendering screen transmitted from the cloud rendering server, including the latency calculation request time and / or latency calculation request identifier, displays the latency information screen in the first display area, and uses the cloud rendering screen as the background of the latency calculation interface.

[0086] The solution in this application embodiment introduces a latency calculation auxiliary program. The streaming service of the cloud rendering server forwards the latency calculation request sent by the client to the latency calculation auxiliary program. This time consumption is almost negligible, reducing the test error introduced by the instruction transmission. The latency calculation request is displayed through the latency calculation auxiliary program, and the displayed latency screen and the cloud rendering screen are sent back to the client for display, so that the client can decode the latency screen and the cloud rendering screen to obtain the accurate end-to-end latency time.

[0087] In this embodiment of the application, when the cloud-rendered screen is displayed as the background of the latency calculation interface, the method further includes: performing character extraction on the latency information screen to obtain character information; obtaining decoding time information when decoding the latency information screen and the cloud-rendered screen transmitted from the cloud; and determining the latency time information between the client and the cloud based on the character information and the decoding time information.

[0088] It should be noted that extracting characters from the delayed information screen can be understood as extracting characters from the overlaid cloud-rendered screen. Since the delayed information screen is superimposed on the cloud-rendered screen, this process involves extracting characters from the superimposed image. The character extraction method can be any method capable of extracting text from the screen; no specific limitation is made here. As an example, Optical Character Recognition (OCR) is used to extract characters from the delayed information screen. The delayed information screen includes the delay calculation request time and / or the delay calculation request identifier. The character information includes the first character corresponding to the delay calculation request time and the second character corresponding to the delay calculation request identifier.

[0089] It should be noted that decoding time can also be understood as the client's response time to the delayed information screen and the cloud-rendered screen. The process involves obtaining the decoding time information when decoding the delayed information screen and the cloud-rendered screen transmitted from the cloud; and determining the latency between the client and the cloud based on the character information and the decoding time information. For example, this can be illustrated by decoding the delayed information screen and the cloud-rendered screen to obtain the decoding time, and then determining the latency based on the decoding time and the character information.

[0090] In this embodiment of the application, the character information includes first character information corresponding to the delay calculation request time and second character information corresponding to the delay calculation request identifier. The process of determining the delay time information between the client and the cloud based on the character information and the decoding time information specifically includes: if the second character information is found in the stored identifier group of the client, the delay time information between the client and the cloud is determined based on the first character information and the decoding time information.

[0091] It should be noted that when the client sends a latency calculation request to the cloud, it caches the corresponding latency calculation flag in a flag group. This flag group deletes flags periodically or at intervals. The interval can be determined based on the actual situation and is not limited here. As an example, the client deletes flags that are 10 seconds (s) older than the specified time in the flag group.

[0092] It should be noted that if the second character information corresponding to the latency calculation identifier is found in the identifier group stored on the client, the latency time is determined based on the second character information corresponding to the latency calculation request time and the decoding time, and the second character information is then deleted from the identifier group. Determining the latency time based on the second character information corresponding to the latency calculation request time and the decoding time can be illustrated by using the difference between the decoding time and the second character information as the latency time.

[0093] It should be noted that if the second character information corresponding to the delay calculation identifier is not found in the identifier group already stored on the client, no further processing will be performed.

[0094] The solution in this application embeds a latency measurement auxiliary program into the application and utilizes a streaming service to collect data from the application, embedding the latency measurement request time into the transmitted frame. Compared to traditional measurement schemes (which add encoded auxiliary frames or auxiliary information), this application reduces the performance overhead of the streaming service. It also avoids the potential latency increase caused by introducing processing logic.

[0095] To facilitate understanding, an example is provided below to illustrate the above solution. Figure 4 This application provides an exemplary flowchart illustrating end-to-end latency in an embodiment of the present application; as shown below. Figure 4 As shown, it includes a client, a cloud rendering example (i.e., the cloud or cloud rendering server mentioned above), and a cloud rendering example including a streaming service, a cloud application (or cloud rendering application), and a latency measurement auxiliary program. The specific steps are as follows:

[0096] 1. Start delay calculation.

[0097] It should be noted that the client sends the instruction to the streaming service to start the delay calculation (i.e., the delay calculation instruction mentioned above).

[0098] 2. Start the delay calculation auxiliary program and set it to the top of the cloud rendering application.

[0099] It should be noted that after receiving the instruction to start latency calculation, the streaming service starts the latency calculation auxiliary program and places it on top of the cloud rendering application.

[0100] 3. Send a delay calculation request.

[0101] It should be noted that the client sends a latency calculation request to the streaming service.

[0102] 4. Forward the delay calculation request.

[0103] It should be noted that the streaming service forwards latency calculation requests to the latency calculation auxiliary program.

[0104] 5. Display the information carried in the delay calculation request.

[0105] It should be noted that the delay calculation auxiliary program displays the information carried in the delay calculation request.

[0106] 6. The request is displayed on the return screen.

[0107] It should be noted that the streaming service sends the drawing back to the client, and the returned image displays the request and the cloud-rendered image.

[0108] 7. Display the response screen and calculate the delay.

[0109] It should be noted that the client will respond with the screen display and calculate the delay.

[0110] For ease of understanding, Figure 5 This application provides another exemplary flowchart illustrating end-to-end latency; as shown below. Figure 5 As shown, the specific steps are as follows:

[0111] 1. The client sends a command to start delay measurement. After receiving and executing the command, the cloud rendering server invokes the delay measurement auxiliary program.

[0112] 2. When a user clicks the "Delay Calculation" button, the client sends a delay calculation request. The delay calculation request includes the button click event, the delay calculation request time, and the delay calculation request ID.

[0113] 3. The streaming service forwards the received latency calculation request time and request ID to the latency calculation assistant program and translates the button click event into a button click on the cloud rendering server.

[0114] 4. After the "Delay Calculation" button of the delay calculation auxiliary program is clicked, the delay calculation request time and delay calculation request ID will be displayed in the request information display area.

[0115] 5. Send the screen containing the request information back to the client as response screen data.

[0116] 6. The client obtains the latency calculation request time from the response screen. The time it takes for the client to decode the response screen is the latency calculation response time. By combining the latency calculation request time and the latency calculation response time, the end-to-end latency can be accurately calculated.

[0117] To make it easier to understand, the above steps are explained in detail below:

[0118] 1. The client has successfully requested the cloud rendering application before starting the latency calculation. When using the real-time cloud rendering application, if the client needs to know the end-to-end latency of the current operation, the client actively sends a latency calculation start command to the cloud rendering server. After receiving the command, the cloud rendering server's streaming service will activate the latency calculation auxiliary program. Figure 6 A schematic diagram of an exemplary delay calculation auxiliary program provided in this application embodiment; as shown Figure 6 As shown, the delay calculation auxiliary program includes a delay calculation button and a request information display area, as detailed below:

[0119] (1) Delay Calculation Button: When the end user clicks this button (in a cloud rendering scenario, clicking the button essentially means clicking the button area displayed on the client screen), a delay calculation request is sent. The delay calculation request contains the button click event, the delay calculation request itself, and the delay calculation request ID. The delay calculation request time is the local system time when the end user clicks the button, accurate to milliseconds. The delay calculation request ID uses an auto-incrementing ID mechanism and is globally unique throughout the entire delay calculation process.

[0120] (2) Request information display information: Dynamically display the terminal latency calculation request time and latency calculation request ID received on the cloud rendering server.

[0121] (3) The delay calculation auxiliary program is displayed above the cloud rendering application and is in the focus state of the window. Figure 7 A schematic diagram illustrating an exemplary delay calculation auxiliary program display method provided in this application embodiment; as shown Figure 7 As shown, the client sends a start delay calculation to the streaming service, which then invokes the delay calculation assistant. The delay calculation assistant is displayed above the cloud rendering application and is in focus. In this state, any control requests translated by the streaming service will be captured by the delay calculation assistant.

[0122] (4) The streaming service and the delay calculation assistant program use kernel-mode inter-process communication (this implementation method is a highly efficient inter-process communication method. Although the streaming service and the delay calculation assistant program are two different processes, the time taken for the streaming service to pass the delay calculation instruction to the delay calculation assistant program is almost negligible after adopting this method, reducing the calculation error introduced by the instruction passing), which is used to forward the delay calculation request time and delay calculation request ID in the delay calculation request to the delay calculation assistant program.

[0123] (5) The image rendered by the end user in real time consists of the cloud rendering application image and the delay calculation auxiliary program image, with the delay calculation auxiliary program image superimposed on the cloud rendering application image.

[0124] 2. When the end user clicks the delay calculation button, a delay calculation request is sent to the streaming service, and the request is cached.

[0125] 3. After receiving the latency calculation request, the streaming service first forwards the latency calculation request time and latency calculation request ID to the latency calculation assistant program; then it translates the button click event into a button click action on the cloud rendering instance.

[0126] 4. After the delay calculation auxiliary program captures the button click action, it displays the delay calculation request time and delay calculation request ID received from the streaming service in the request information display area. Figure 8This application provides an exemplary schematic diagram showing the delay calculation request time and delay calculation request identifier displayed in a delay calculation auxiliary program; as shown. Figure 8 As shown, the delay calculation request time and delay calculation request identifier are displayed in the request information display area of ​​the delay calculation auxiliary program.

[0127] The delay calculation auxiliary program displays the information sent by the streaming service, not based on the time it receives the information, but on the time it captures the button click action. This allows for the calculation of the control command translation and response time. The response actions to delay calculation requests have a clear sequence.

[0128] 5. Send the screen containing the request information back to the terminal as response screen data. Figure 9 This application provides an exemplary schematic diagram illustrating the process by which a cloud rendering server sends a response screen containing request information back to the client; for example, Figure 9 As shown, the streaming service will send the request information (including the latency calculation request time and latency calculation request ID) and the cloud-rendered image back to the client as a media stream for display.

[0129] 6. The terminal obtains the latency calculation request time from the response screen, and the time it takes for the client to decode the response screen is the latency calculation response time. By combining the latency calculation response time and the latency calculation request time, the end-to-end latency can be accurately calculated: End-to-end latency = Latency calculation response time - Latency calculation request time. Figure 10 A schematic diagram illustrating the extraction of delay measurement request time as provided in an embodiment of this application; such as Figure 10 As shown, the images of the screen with request information and the cloud rendering screen are cropped to obtain the screen with request information. OCR recognition is then performed on the screen with request information to obtain the character information corresponding to the request information, including the character information corresponding to the delay calculation time and the delay request ID.

[0130] (1) In order to accurately extract the information of the requested information display area on the terminal, before the image is OCR recognized, it is also necessary to extract the image of the information display area according to the characteristics of the delay calculation auxiliary program (first identify the position of the delay calculation button using OpenCV, and then extract the image of the information display area relative to the calculation button according to the position of the information display area).

[0131] (2) Use OCR technology to extract the delay calculation request time and delay calculation request ID of the information display area.

[0132] (3) Use the delay calculation request ID to query the above cached delay calculation request cache.

[0133] The purpose of the latency measurement request ID is to allow the terminal to filter out expired latency measurement requests, improving the accuracy of the solution. This is because the streaming service frequently captures images from the latency measurement auxiliary program in real time. However, the latency measurement requests issued by the user are random. Therefore, the content of the information display area will be the same in N consecutive image frames sent by the cloud rendering server.

[0134] If a request is found, the current record in the cache is deleted first. Then, the time it takes for the current client to decode the screen is the latency-calculated response time. The result of subtracting the latency-calculated request time from the latency-calculated response time is the end-to-end latency (both the latency-calculated response time and the latency-calculated request time are the client's local time and have the same time baseline, so subtracting the two gives the accurate end-to-end latency).

[0135] If the request is not found, it means that the request has expired, and you can simply ignore the data in the current group.

[0136] This application provides an end-to-end delay measurement device. Figure 11 This is a schematic diagram of the structure of an end-to-end delay measurement device provided in an embodiment of this application; as shown below. Figure 11 As shown, the end-to-end latency measurement device 1100, applied in the cloud, includes:

[0137] The startup unit 1101 is used to respond to the latency calculation command received from the client, start the latency calculation auxiliary program and place it on top of the cloud rendering application;

[0138] The forwarding unit 1102 is used to obtain the delay calculation request sent by the client and forward the delay calculation request to the delay calculation auxiliary program; the delay calculation request is generated by the client in response to the touch operation of the first delay calculation button in the delay calculation interface;

[0139] The feedback unit 1103 is used to acquire the delay information screen and cloud rendering screen displayed by the delay calculation auxiliary program and send them back to the client for display.

[0140] Optionally, the delay calculation request includes a button touch command; after forwarding the delay calculation request to the delay calculation auxiliary program, the end-to-end delay calculation device 1100 further includes a triggering unit, which translates the button touch command and triggers the delay calculation auxiliary program to display the delay information screen.

[0141] Optionally, the delay calculation request may further include at least one of the following: delay calculation request time and delay calculation request identifier; the triggering unit is also used to trigger the delay calculation auxiliary program to display the delay calculation request time and / or the delay calculation request identifier as the delay information screen.

[0142] Based on the hardware implementation of the above program modules, and in order to implement the cloud-side method of this application embodiment, this application embodiment also provides an electronic device. Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 12 As shown, the electronic device 1200 includes: a first processor 1201 and a first memory 1203 for storing computer programs that can run on the first processor 1201. Optionally, the electronic device 1200 may also include a first communication bus 1202.

[0143] In specific embodiments, the first processor 1201 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the first processor can also be other types, and this embodiment does not impose specific limitations.

[0144] In this embodiment, the first communication bus 1202 is used to establish communication between the first processor 1201 and the first memory 1203; when the first processor 1201 executes the running program stored in the first memory 1203, it implements the following end-to-end delay calculation method:

[0145] In response to the latency calculation command initiated by the client, the latency calculation auxiliary program is launched and placed on top of the cloud rendering application; the latency calculation request sent by the client is obtained and forwarded to the latency calculation auxiliary program; the latency calculation request is generated by the client in response to the touch operation of the first latency calculation button in the latency calculation interface; the latency information screen and cloud rendering screen displayed by the latency calculation auxiliary program are obtained and sent back to the client for display.

[0146] Furthermore, the delay calculation request includes a button touch instruction; after forwarding the delay calculation request to the delay calculation auxiliary program, the first processor 1201 is further configured to translate the button touch instruction and trigger the delay calculation auxiliary program to display the delay information screen.

[0147] Furthermore, the delay calculation request also includes at least one of the following: delay calculation request time and delay calculation request identifier; the first processor 1201 is also used to trigger the delay calculation auxiliary program to display the delay calculation request time and / or the delay calculation request identifier as the delay information screen.

[0148] This application also provides an end-to-end delay measurement device. Figure 13 This is a schematic diagram of another end-to-end delay measurement device provided in an embodiment of this application; as shown below. Figure 13 As shown, the end-to-end latency measurement device 1300, applied to the client side, includes:

[0149] Display unit 1301 is used to display a delay calculation interface in response to a start delay calculation command; the delay calculation interface includes a first delay calculation button and a first display area;

[0150] The generation unit 1302 is used to generate a delay calculation request and send it to the cloud in response to a touch operation on the first delay calculation button in the delay calculation interface;

[0151] The display unit 1303 is used to respond to the latency information screen and cloud rendering screen transmitted from the cloud, display the latency information screen in the first display area, and display the cloud rendering screen as the interface background of the latency calculation interface.

[0152] Optionally, the display unit 1303 is further configured to display the delay information screen, including the delay calculation request time and / or the delay calculation request identifier, in the first display area.

[0153] Optionally, after displaying the cloud-rendered image as the background of the latency measurement interface, the end-to-end latency measurement device 1300 further includes:

[0154] A character extraction unit is used to perform character extraction operations on the delayed information screen to obtain character information;

[0155] The decoding unit is used to obtain the decoding time information when decoding the delayed information image and cloud rendering image transmitted back from the cloud.

[0156] The determining unit is used to determine the delay time information between the client and the cloud based on the character information and the decoding time information.

[0157] Optionally, the character information includes first character information corresponding to the delay calculation request time and second character information corresponding to the delay calculation request identifier; the determining unit is further configured to determine the delay time information between the client and the cloud based on the first character information and the decoding time information when the second character information is found in the stored identifier group of the client.

[0158] Based on the hardware implementation of the above program modules, and in order to implement the method on the adjacent client side of the embodiments of this application, the embodiments of this application provide an electronic device. Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 14 As shown, the electronic device 1400 includes a second processor 1401 and a second memory 1403 for storing computer programs that can run on the second processor 1401. Optionally, the electronic device 1400 also includes a second communication bus 1402.

[0159] In specific embodiments, the second processor 1401 can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this embodiment does not impose specific limitations.

[0160] In this embodiment, the second communication bus 1402 is used to realize the connection communication between the second processor 1401 and the second memory 1403; when the second processor 1401 executes the running program stored in the second memory 1403, it implements the following end-to-end delay calculation method:

[0161] In response to the start delay calculation command, a delay calculation interface is displayed; the delay calculation interface includes a first delay calculation button and a first display area; in response to a touch operation on the first delay calculation button in the delay calculation interface, a delay calculation request is generated and sent to the cloud; in response to the delay information screen and cloud rendering screen transmitted back from the cloud, the delay information screen is displayed in the first display area, and the cloud rendering screen is used as the interface background of the delay calculation interface.

[0162] Furthermore, the second processor 1401 is also used to display the delay information screen, including the delay calculation request time and / or the delay calculation request identifier, in the first display area.

[0163] Furthermore, the second processor 1401 is also used to perform character extraction operations on the delayed information screen to obtain character information; obtain decoding time information when decoding the delayed information screen and cloud rendering screen transmitted from the cloud; and determine the delay time information between the client and the cloud based on the character information and the decoding time information.

[0164] Furthermore, the character information includes first character information corresponding to the delay calculation request time and second character information corresponding to the delay calculation request identifier; the second processor 1401 is also used to determine the delay time information between the client and the cloud based on the first character information and the decoding time information when the second character information is found in the stored identifier group of the client.

[0165] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the end-to-end latency calculation method on the cloud side as described above, or implements the end-to-end latency calculation method on the client side as described above.

[0166] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors. The computer program implements the end-to-end latency calculation method on the cloud side as described above, or implements the end-to-end latency calculation method on the client side as described above.

[0167] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0169] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A method for measuring end-to-end delay, characterized in that, The method is applied to the cloud end, and the method comprises: In response to an acquired client-initiated delay measurement instruction, a delay measurement auxiliary program is started and is pinned on a cloud rendering application; An acquired delay measurement request sent by the client is forwarded to the delay measurement auxiliary program; the delay measurement request is generated by the client in response to a touch operation on a first delay measurement button in a delay measurement interface; Delay information screens and cloud rendering screens displayed by the delay measurement auxiliary program are acquired and are returned to the client for display.

2. The method of claim 1, wherein, The delay measurement request comprises a button touch instruction; after the delay measurement request is forwarded to the delay measurement auxiliary program, the method further comprises: After the button touch instruction is translated, the delay measurement auxiliary program is triggered to display the delay information screens.

3. The method of claim 2, wherein, The delay measurement request further comprises at least one of a delay measurement request time and a delay measurement request identifier; the triggering of the delay measurement auxiliary program to display the delay information screens comprises: The delay measurement auxiliary program is triggered to display the delay measurement request time and / or the delay measurement request identifier as the delay information screens.

4. A method of estimating end-to-end delay, the method comprising: The method is applied to the client, and the method comprises: In response to a started delay measurement instruction, a delay measurement interface is displayed; the delay measurement interface comprises a first delay measurement button and a first display area; In response to a touch operation on the first delay measurement button in the delay measurement interface, a delay measurement request is generated and is sent to the cloud end; In response to delay information screens and cloud rendering screens returned by the cloud end, the delay information screens are displayed in the first display area, and the cloud rendering screens are displayed as interface backgrounds of the delay measurement interface.

5. The method of claim 4, wherein, The display of the delay information screens in the first display area comprises: The delay information screens comprising a delay measurement request time and / or a delay measurement request identifier are displayed in the first display area.

6. The method of claim 4, wherein, After the cloud rendering screens are displayed as interface backgrounds of the delay measurement interface, the method further comprises: Character information is obtained by performing a character extraction operation on the delay information screens; Decoding time information is acquired when the delay information screens and the cloud rendering screens returned by the cloud end are decoded; Delay time information between the client and the cloud end is determined according to the character information and the decoding time information.

7. The method of claim 6, wherein, The character information comprises first character information corresponding to the delay measurement request time and second character information corresponding to the delay measurement request identifier; The determination of the delay time information between the client and the cloud end according to the character information and the decoding time information comprises: In a case where the second character information is found in a stored identifier group of the client, the delay time information between the client and the cloud end is determined according to the first character information and the decoding time information.

8. An electronic device, comprising: The method comprises: a processor and a memory for storing a computer program capable of running on the processor, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3, or to implement the steps of the method of any one of claims 4 to 7.

9. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3, or to implement the steps of the method of any one of claims 4 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3, or to implement the steps of the method of any one of claims 4 to 7.

Images