Data processing method and device, head-mounted display device, and medium
By acquiring the latency parameter information of data frames to perform streaming performance analysis, the latency problem in the streaming process was solved, and the user experience was improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GOERTEK INC
- Filing Date
- 2023-04-14
- Publication Date
- 2026-06-05
AI Technical Summary
The latency issue during streaming has not been effectively resolved, impacting the user experience.
By acquiring the latency parameters of data frames, including encoding duration, transmission duration, and decoding duration, streaming performance analysis can be performed for targeted optimization.
It improves the performance of the streaming process and enhances the user experience.
Smart Images
Figure CN116567289B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of wearable device technology, and more specifically, to a data processing method, a data processing apparatus, a head-mounted display device, and a computer-readable storage medium. Background Technology
[0002] Streaming typically refers to transmitting the content displayed on the screen of one electronic device to the screen of another electronic device for display. This process involves capturing, encoding, transmitting, decoding, and displaying audio and video.
[0003] Latency during streaming is a core indicator for evaluating the quality of streaming functionality. Therefore, it is necessary to statistically analyze the latency at each stage of the entire streaming process to facilitate optimization of the streaming functionality. Summary of the Invention
[0004] One objective of this disclosure is to provide a new technical solution for data processing.
[0005] According to a first aspect of the present disclosure, a data processing method is provided, the method comprising:
[0006] When the first module transmits a data frame to the second module via a streaming connection, the latency parameter information of the data frame is obtained; wherein, the latency parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame;
[0007] Based on the aforementioned delay parameter information, the streaming performance analysis results are obtained.
[0008] Optionally, outgoing data corresponding to the data frame is obtained; wherein, the outgoing data has the encoded data frame;
[0009] Identify the outgoing data and obtain the latency parameter information of the data frame.
[0010] Optionally, the delay parameter information includes the encoding duration, transmission duration, and decoding duration of the data frame.
[0011] The step of identifying the outgoing data and obtaining the latency parameter information of the data frame includes:
[0012] Obtain the encoding start time and encoding end time corresponding to the data frame in the outgoing data;
[0013] The encoding duration corresponding to the data frame is obtained based on the encoding start time and the encoding end time;
[0014] Obtain the start transmission time and the receiving time of the data frame corresponding to the outgoing data;
[0015] The transmission duration corresponding to the data frame is obtained based on the start transmission time and the receiving time of the peer end;
[0016] Obtain the peer's start decoding time and peer's completion decoding time corresponding to the data frame in the outgoing data;
[0017] The decoding duration corresponding to the data frame is obtained based on the peer's start decoding time and the peer's completion decoding time.
[0018] Optionally, the method further includes the step of generating outgoing data corresponding to the data frame.
[0019] The generation of the outgoing data corresponding to the data frame includes:
[0020] When the data frame is captured by the first module, a first storage area is requested to record the outgoing data corresponding to the data frame, and the time of requesting the first storage area is recorded in the encoding start timestamp field of the outgoing data to obtain the encoding start time corresponding to the data frame, and the encoding of the data frame begins.
[0021] When the data frame is encoded, the time when the encoding of the data frame is completed is recorded in the encoding end timestamp field of the outgoing data to obtain the encoding end time corresponding to the data frame, and the encoded data frame is recorded in the encoded data field of the outgoing data.
[0022] When an encoded data frame is sent from the first module to the second module, the time of sending the encoded data frame is recorded in the start transmission timestamp field of the outgoing data to obtain the start transmission time corresponding to the data frame.
[0023] Upon receiving the encoded data frame sent by the first module, the time of receiving the encoded data frame is recorded in the peer-received timestamp field of the outgoing data to obtain the peer-received time corresponding to the data frame.
[0024] When decoding the encoded data frame sent by the first module is started, the time when decoding of the encoded data frame begins is recorded in the peer-end start decoding timestamp field of the outgoing data to obtain the peer-end start decoding time corresponding to the data frame.
[0025] When the encoded data frame is decoded, the time when the decoding of the encoded data frame is completed is recorded in the peer decoding completion timestamp field of the outgoing data to obtain the peer decoding completion time corresponding to the data frame.
[0026] The latency parameter information also includes the total latency corresponding to the data frame;
[0027] The total latency corresponding to the data frame is calculated based on the encoding duration, transmission duration, and decoding duration of the data frame.
[0028] Optionally, the method further includes:
[0029] When the first module transmits data frames to the second module via a streaming connection, a first canvas is created in the desktop environment;
[0030] The transmitted data frames are rendered and displayed using the first canvas.
[0031] According to a second aspect of the present disclosure, a data processing apparatus is provided, the apparatus comprising:
[0032] Create a module to create the first canvas in the desktop environment of the first device;
[0033] The first acquisition module is used to acquire the delay parameter information of the data frame when the first module transmits the data frame to the second module through a streaming connection; wherein the delay parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame.
[0034] The second acquisition module is used to obtain the streaming performance analysis results based on the delay parameter information.
[0035] Optionally, the first acquisition module is specifically used for:
[0036] Obtain the outgoing data corresponding to the data frame; wherein the outgoing data has the encoded data frame;
[0037] Identify the outgoing data and obtain the latency parameter information of the data frame.
[0038] According to a third aspect of the present disclosure, a head-mounted display device is provided, the head-mounted display device comprising: a memory for storing executable computer instructions; and a processor for executing the interactive control method according to the first aspect above, under the control of the executable computer instructions.
[0039] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, perform the interactive control method described in the first aspect above.
[0040] One beneficial effect of this embodiment is that when the first module transmits a data frame to the second module through a streaming connection, it obtains the latency parameter information of the data frame. The latency parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame. Then, the streaming performance analysis result is obtained based on the latency parameter information. In this way, targeted optimization can be performed based on the streaming performance analysis result to improve the user experience.
[0041] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.
[0043] Figure 1 This is a schematic diagram of the hardware configuration of a data processing system according to an embodiment of the present disclosure;
[0044] Figure 2 This is a schematic flowchart of a data processing method according to an embodiment of the present disclosure;
[0045] Figure 3 This is a schematic diagram illustrating the composition of outgoing data according to an embodiment of this disclosure;
[0046] Figure 4 This is a schematic block diagram of a data processing apparatus according to embodiments of the present disclosure;
[0047] Figure 5 This is a schematic block diagram of a head-mounted display device according to an embodiment of the present disclosure. Detailed Implementation
[0048] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.
[0049] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0050] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0051] In all the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0052] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0053] <Hardware Configuration>
[0054] Figure 1 This is a schematic diagram of the hardware configuration of a data processing system that can be used to implement a data processing method of one embodiment. Figure 1 A first device 100, a second device 200, and a network 300 are illustrated. The first device 100 can connect to the network 300 and can also connect to the second device 200 via a communication method such as Bluetooth. In one embodiment, the first device 100 connects to the second device 200 only via a communication method such as Bluetooth. Multiple servers 301 and 302 can be configured in the network 300. The network 300 can be a wireless communication network or a wired communication network. The network 300 can be a local area network (LAN) or a wide area network (WAN). The network 300 can be for short-range communication or long-range communication.
[0055] In one embodiment, such as Figure 1 As shown, the first device 100 may include a processor 101 and a memory 102. The first device 100 also includes a communication device 103, a display device 104, a user interface 105, a camera device 106, an audio / video interface 107, and a sensor 108, etc. Furthermore, the first device 100 may also include a power management chip 109 and a battery 110, etc.
[0056] The first device 100 may be a mobile phone, a portable computer, a tablet computer, a handheld computer, a wearable device, etc., and this disclosure does not limit it. Figure 1 The components shown are merely illustrative. The first device 100 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The first device 100 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.
[0057] In one embodiment, such as Figure 1As shown, the second device 200 may include a processor 201 and a memory 202. The second device 200 also includes a communication device 203, a display device 204, a user interface 205, a camera device 206, an audio / video interface 207, and a sensor 208, etc. Furthermore, the second device 200 may also include a power management chip 209 and a battery 210, etc.
[0058] The processor 201 can be any type of processor. The memory 202 can store the underlying software, system software, application software, data, etc., required for the operation of the second device 200. The memory 202 can include various forms of memory, such as ROM, RAM, Flash, etc. The communication device 203 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, or 5G communication device. Through the communication device 203, the second device 200 can be deployed in a network. The display device 204 can be a liquid crystal display, an OLED display, etc. In one example, the display device 204 can be a touchscreen. Users can perform input operations through the display device 204. Furthermore, users can also perform fingerprint recognition, etc., through the touchscreen. The user interface 205 can include a USB interface, a Lightning interface, a keyboard, etc. The camera device 206 can be a single camera or multiple cameras. The audio / video interface 207 can include, for example, a speaker interface, a microphone interface, a video transmission interface such as HDMI, etc. The sensor 208 can include, for example, a gyroscope, an accelerometer, a temperature sensor, a humidity sensor, a pressure sensor, etc. For example, the sensor can determine the attitude information of the second device. The power management chip 209 can be used to manage the power supply of the second input device 200, and can also manage the battery 210 to ensure maximum utilization efficiency. The battery 210 is, for example, a lithium-ion battery.
[0059] The second device 200 may be a head-mounted display device. For example, AR (Augmented Reality) glasses and MR (Mixed Reality) glasses, etc., but this disclosure does not limit the scope of the embodiments. Figure 1 The components shown are merely illustrative. The second device 200 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The second device 200 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.
[0060] In this embodiment, the memory 202 of the second device 200 is used to store program instructions that control the processor 201 to perform data processing methods. Those skilled in the art can design these instructions based on the disclosed scheme of this invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.
[0061] It should be understood that, despite Figure 1 Only one first device 100 and one second device 200 are shown, but this does not mean that the number of each is limited. The data processing system may contain multiple first devices 100 and multiple second devices 200.
[0062] In the above description, those skilled in the art can design instructions based on the solutions provided in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.
[0063] <Method Implementation>
[0064] Figure 2 An embodiment of the present disclosure illustrates a data processing method that can be implemented by a head-mounted display device, or by a control device independent of the head-mounted display device and the head-mounted display device together, or by a cloud server and the head-mounted display device together.
[0065] like Figure 2 As shown, the data processing method of this embodiment may include the following steps S2100 to S2300:
[0066] Step S2100: When the first module transmits a data frame to the second module via a streaming connection, the delay parameter information of the data frame is obtained.
[0067] The first module is typically located in a first device, such as a mobile phone, and the second module is typically located in a second device, such as a head-mounted display device.
[0068] In this embodiment, a streaming connection is established between the first module and the second module. This streaming connection can be a wireless streaming connection or a wired streaming connection. During the operation of the 3D desktop environment, the second module creates multiple canvases in the 3D desktop environment and establishes a correspondence between the canvases and the virtual screens created by the first module. Then, it displays the data frames of the virtual screens of the first module through the corresponding canvases.
[0069] Specifically, when the first module transmits data frames to the second module via a streaming connection, the second module creates a first canvas in the desktop environment and renders and displays the data frames transmitted by the first module through the first canvas. The first canvas is one of multiple canvases created by the second module, used to render and display the data frames of a first virtual screen transmitted by the first module to the second module via the streaming connection. The first virtual screen runs a first application.
[0070] For example, when the virtual screen 1 of the first module runs a game application, the first module will capture the data frames displayed on the virtual screen 1 and transmit the captured data frames of the virtual screen 1 to the second module through a streaming connection. The second module will then render and display the data frames of the virtual screen 1 on the canvas 1.
[0071] More specifically, the first module typically captures data frames from the first virtual screen, encodes these frames to obtain encoded data frames, and then transmits them to the second module via a streaming connection. The second module receives the encoded data frames, decodes them, and then renders and displays the decoded data frames through the first canvas. In other words, the rendering and displaying of the transmitted data frames through the first canvas mentioned in this step refers to rendering and displaying the decoded data frames through the first canvas.
[0072] It should be noted that the data frame before decoding and the data frame after decoding are usually the same; that is, decoding and encoding are a pair of inverse operations.
[0073] The delay parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame.
[0074] Optionally, the latency parameter information may further include the total latency corresponding to the data frame; wherein the total latency corresponding to the data frame is calculated based on the encoding duration, transmission duration, and decoding duration corresponding to the data frame. Specifically, the total latency T0 corresponding to the data frame satisfies the following formula:
[0075] T0 = T1 + T2 + T3 (1)
[0076] In the above formula (1), T1 is the encoding duration of the data frame, T2 is the transmission duration of the data frame, and T3 is the decoding duration of the data frame.
[0077] In this embodiment, when the first module transmits a data frame to the second module via a streaming connection, the second module will obtain the latency parameter information of the data frame.
[0078] In a specific embodiment, step S2200, obtaining the delay parameter information of the data frame, may further include the following steps S2210 to S2220:
[0079] Step S2210: Obtain the outgoing data corresponding to the data frame.
[0080] The outgoing data consists of encoded data frames and has a specific data format.
[0081] Specifically, when the second module renders and displays the decoded data frame of the first virtual screen through the first canvas, it will obtain the outgoing data corresponding to the data frame.
[0082] In this embodiment, to support the calculation of streaming delay, a custom data format is defined between the first and second modules to record outgoing data with encoded data frames. Part of the information in this outgoing data is filled in on the first module side, as shown in the reference... Figure 3 For example, the encoding start time of the data frame, the encoding end time of the data frame, the encoded data frame, and the start transmission time of the data frame. Other information in this outgoing data is filled in on the second module side, see [reference]. Figure 3 For example, the receiving time of the data frame at the peer end, the start decoding time of the data frame at the peer end, and the completion decoding time of the data frame at the peer end. Furthermore, the second module is also used to parse the outgoing data, thereby calculating the delay parameter information of the data frame. Here, before performing step S2210 to obtain the outgoing data corresponding to the data frame, the data processing method of this embodiment further includes a step of generating the outgoing data corresponding to the data frame. Generating the outgoing data corresponding to the data frame may further include the following steps S3100 to S3600:
[0083] Step S3100: When the data frame is captured by the first module, a first storage area is requested to record the outgoing data corresponding to the data frame, and the time of requesting the first storage area is recorded in the encoding start timestamp field of the outgoing data to obtain the encoding start time corresponding to the data frame, and the encoding of the data frame begins.
[0084] For those who can understand, please refer to Figure 3 To facilitate recording the outgoing data of each data frame, a data frame ID field is added to the outgoing data. When transmitting the first data frame of the first virtual screen, the data frame ID field is 1, and the data frame ID field is incremented by 1 for each data frame transmitted.
[0085] In this embodiment, after the first module captures the data frame of the first virtual screen, it requests a storage area according to the aforementioned custom data format to record the outgoing data corresponding to the data frame. Typically, the custom data format includes encoded data fields, a data frame ID field, an encoding start timestamp field, an encoding end timestamp field, a transmission start timestamp field, a peer-received timestamp field, a peer-decoding start timestamp field, and a peer-completed decoding timestamp field. That is, the outgoing data includes the values of the encoded data fields, the data frame ID field, the encoding start timestamp field, the encoding end timestamp field, the transmission start timestamp field, the peer-received timestamp field, the peer-decoding start timestamp field, and the peer-completed decoding timestamp field. Initially, the values of all fields in this outgoing data are empty.
[0086] Typically, when the first module requests the first storage area, it records the time of requesting the first storage area as the value of the encoding start timestamp field in the outgoing data, thereby obtaining the encoding start time corresponding to the data frame, and starting to encode the data frame.
[0087] Step S3200: When the data frame is encoded, the time when the encoding of the data frame is completed is recorded in the encoding end timestamp field of the outgoing data to obtain the encoding end time corresponding to the data frame, and the encoded data frame is recorded in the encoded data field of the outgoing data.
[0088] In this embodiment, when the first module completes the encoding of the data frame and obtains the encoded data frame, it records the time when the data frame is completed as the value of the encoding end timestamp field in the outgoing data, thereby obtaining the encoding end time corresponding to the data frame, and at the same time records the encoded data frame in the encoded data field of the outgoing data.
[0089] Step S3300: When sending an encoded data frame to the second module through the first module, the time of sending the encoded data frame is recorded in the start transmission timestamp field of the outgoing data to obtain the start transmission time corresponding to the data frame.
[0090] In this embodiment, the first module transmits outgoing data to the second module via a streaming connection, and records the time of sending the encoded data frame as the value of the start transmission timestamp field in the outgoing data, thereby obtaining the start transmission time corresponding to the data frame. Simultaneously, the first module increments the value of the data frame ID field in the outgoing data by 1.
[0091] Step S3400: Upon receiving the encoded data frame sent by the first module, the time of receiving the encoded data frame is recorded in the peer-received timestamp field of the outgoing data to obtain the peer-received time corresponding to the data frame.
[0092] In this embodiment, the second module receives outgoing data sent by the first module, wherein the outgoing data has encoded data frames. After receiving the outgoing data, the second module records the time of receiving the outgoing data as the value of the peer-received timestamp field in the outgoing data, thereby obtaining the peer-received time corresponding to the data frame.
[0093] Step S3500: When decoding the encoded data frame sent by the first module is started, the time when decoding of the encoded data frame starts is recorded in the peer start decoding timestamp field of the outgoing data to obtain the peer start decoding time corresponding to the data frame.
[0094] In this embodiment, the second module starts decoding the encoded data frame and records the time when decoding the encoded data frame begins as the peer's decoding start timestamp field in the outgoing data, thereby obtaining the peer's decoding start time corresponding to the data frame.
[0095] Step S3600: If the encoded data frame has been decoded, the time when the decoding of the encoded data frame is completed is recorded in the peer decoding completion timestamp field of the outgoing data to obtain the peer decoding completion time corresponding to the data frame.
[0096] In this embodiment, when the second module completes the decoding of the encoded data frame, it records the time when the decoding of the encoded data frame is completed as the value of the peer decoding completion timestamp field in the outgoing data, thereby obtaining the peer decoding completion time corresponding to the data frame. At this point, the values of all fields in the outgoing data are filled.
[0097] Based on steps S3100 to S3700 above, the outgoing data corresponding to the data frame can be generated by the first module and the second module. Then, the second module parses the data frame to obtain the delay parameter information of the data frame.
[0098] Specifically, the second module can determine the outgoing data corresponding to the current data frame based on the value of the data frame ID field in the received outgoing data.
[0099] Step S2220: Identify the outgoing data and obtain the latency parameter information of the data frame.
[0100] Specifically, step S2220, which identifies the outgoing data and obtains the delay parameter information of the data frame, may further include the following steps S2221 to S2226:
[0101] Step S2221: Obtain the encoding start time and encoding end time corresponding to the data frame in the outgoing data.
[0102] Step S2222: Obtain the encoding duration of the data frame based on the encoding start time and the encoding end time.
[0103] Step S2223: Obtain the start transmission time and the receiving time of the data frame corresponding to the outgoing data.
[0104] Step S2224: Obtain the transmission duration of the data frame based on the start transmission time and the receiving time of the peer.
[0105] Step S2225: Obtain the peer's start decoding time and peer's completion decoding time corresponding to the data frame in the outgoing data.
[0106] Step S2226: Obtain the decoding duration of the data frame based on the decoding start time and decoding completion time of the peer.
[0107] According to an embodiment of this disclosure, when the first module transmits a data frame to the second module via a streaming connection, it acquires the latency parameter information of the data frame. The latency parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame. Then, the streaming performance analysis result is obtained based on the latency parameter information. In this way, targeted optimization can be performed based on the streaming performance analysis result to improve the user experience.
[0108] <Example>
[0109] The following example, using a mobile phone as the first device and AR glasses as the second device, illustrates a data processing method that may include the following steps:
[0110] Step 401: The mobile phone creates a virtual screen 1 and runs application 1 on the virtual screen 1.
[0111] Step 402: The mobile phone captures the data frame of virtual screen 1, applies for the first storage area to record the outgoing data corresponding to the data frame of virtual screen 1, and records the time of applying for the first storage area into the encoding start timestamp field in the outgoing data to obtain the encoding start time corresponding to the data frame. At the same time, the captured data frame of virtual screen 1 is encoded.
[0112] Step 403: When the mobile phone completes the encoding of the data frame of the captured virtual screen 1, it will record the time of completion of encoding of the data frame in the encoded data field of the outgoing data to obtain the encoding end time corresponding to the data frame, and at the same time record the encoded data frame in the encoded data field of the outgoing data.
[0113] Step 404: Based on the wireless streaming connection between the mobile phone and the AR glasses, the mobile phone sends outgoing data to the AR glasses and records the time of sending the outgoing data in the start transmission timestamp field of the outgoing data to obtain the start transmission time corresponding to the data frame. At the same time, the value of the data frame ID field in the outgoing data is incremented by 1.
[0114] Step 405: After the AR glasses receive the outgoing data, they will record the time of receiving the outgoing data in the peer-received timestamp field of the outgoing data to obtain the peer-received time corresponding to the data frame.
[0115] Step 406: The AR glasses begin decoding the encoded data frame of the outgoing data and record the time when decoding of the encoded data frame begins in the peer-end decoding start timestamp field of the outgoing data to obtain the peer-end decoding start time corresponding to the data frame.
[0116] Step 407: After the AR glasses have completed decoding the encoded data frame, they record the time when the decoding of the encoded data frame is completed into the peer decoding timestamp field in the outgoing data to obtain the peer decoding time corresponding to the data frame.
[0117] Step 408: The AR glasses create canvas 1 and render the decoded data frame on canvas 1. At the same time, the value of the outgoing data encoding start timestamp field is extracted to the value of the decoding completion timestamp field of the other end.
[0118] Step 409: Obtain the encoding duration of the data frame based on the encoding start time and encoding end time in the outgoing data; obtain the transmission duration of the data frame based on the transmission start time and the receiving time of the peer in the outgoing data; obtain the decoding duration of the data frame based on the decoding start time and the decoding completion time of the peer; and obtain the total latency of the data frame based on the encoding duration, transmission duration, and decoding duration.
[0119] Step 410: Obtain the streaming performance analysis results based on the delay parameter information.
[0120] <Device Embodiment>
[0121] Figure 4 This is a schematic diagram of a data processing apparatus according to one embodiment, with reference to... Figure 4As shown, the device 400 includes a first acquisition module 410 and a second acquisition module 420.
[0122] The first acquisition module 410 is used to acquire the delay parameter information of the data frame when the first module transmits the data frame to the second module through a streaming connection; wherein the delay parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame.
[0123] The second acquisition module 420 is used to obtain the streaming performance analysis results based on the delay parameter information.
[0124] In one embodiment, the first acquisition module 410 is specifically used to: acquire outgoing data corresponding to the data frame; wherein the outgoing data has the encoded data frame; identify the outgoing data, and obtain the delay parameter information of the data frame.
[0125] In one embodiment, the latency parameter information includes the encoding duration, transmission duration, and decoding duration of the data frame. The first acquisition module 410 is specifically configured to: acquire the encoding start time and encoding end time corresponding to the data frame in the outgoing data; obtain the encoding duration corresponding to the data frame based on the encoding start time and encoding end time; acquire the start transmission time and peer reception time corresponding to the data frame in the outgoing data; obtain the transmission duration corresponding to the data frame based on the start transmission time and peer reception time; acquire the peer start decoding time and peer completion decoding time corresponding to the data frame in the outgoing data; and obtain the decoding duration corresponding to the data frame based on the peer start decoding time and peer completion decoding time.
[0126] In one embodiment, the apparatus further includes a generation module (not shown in the figure), configured to: when the data frame is captured by the first module, allocate a first storage area to record the outgoing data corresponding to the data frame, and record the time of allocating the first storage area in the encoding start timestamp field of the outgoing data to obtain the encoding start time corresponding to the data frame, and start encoding the data frame; when the encoding of the data frame is completed, record the time of completion of encoding of the data frame in the encoding end timestamp field of the outgoing data to obtain the encoding end time corresponding to the data frame, and record the encoded data frame in the encoded data field of the outgoing data; when the encoded data frame is sent to the second module through the first module, record the time of sending the encoded data frame... The start transmission timestamp field of the outgoing data is recorded to obtain the start transmission time corresponding to the data frame; when the encoded data frame sent by the first module is received, the time of receiving the encoded data frame is recorded in the peer-received timestamp field of the outgoing data to obtain the peer-received time corresponding to the data frame; when decoding the received encoded data frame sent by the first module begins, the time of starting decoding of the encoded data frame is recorded in the peer-decoding start timestamp field of the outgoing data to obtain the peer-decoding start time corresponding to the data frame; when decoding of the encoded data frame is completed, the time of completing decoding of the encoded data frame is recorded in the peer-decoding completion timestamp field of the outgoing data to obtain the peer-decoding completion time corresponding to the data frame.
[0127] In one embodiment, the latency parameter information further includes the total latency corresponding to the data frame;
[0128] The total latency corresponding to the data frame is calculated based on the encoding duration, transmission duration, and decoding duration of the data frame.
[0129] In one embodiment, the device 400 further includes a creation module and a display module (not shown in the figure).
[0130] Create a module to create a first canvas in the desktop environment when the first module transmits data frames to the second module via a streaming connection;
[0131] The display module is used to render and display the transmitted data frames through the first canvas.
[0132] According to an embodiment of this disclosure, when the first module transmits a data frame to the second module via a streaming connection, it acquires the latency parameter information of the data frame. The latency parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame. Then, the streaming performance analysis result is obtained based on the latency parameter information. In this way, targeted optimization can be performed based on the streaming performance analysis result to improve the user experience.
[0133] <Equipment Example>
[0134] Figure 5 This is a schematic diagram of the hardware structure of a head-mounted display device according to one embodiment. Figure 5 As shown, the head-mounted display device 500 includes a processor 510 and a memory 520.
[0135] The memory 520 can be used to store executable computer instructions.
[0136] The processor 510 can be used to execute the data processing method according to the method embodiments of this disclosure, under the control of the executable computer instructions.
[0137] The head-mounted display device 500 can be as follows: Figure 1 The second device 200 shown can also be a device with other hardware structures, which is not limited here.
[0138] In another embodiment, the head-mounted display device 500 may include the above-mentioned interactive control device 400.
[0139] In one embodiment, each module of the above-mentioned interactive control device 400 can be implemented by the processor 510 running computer instructions stored in the memory 520.
[0140] Computer-readable storage media
[0141] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the data processing method provided in this disclosure.
[0142] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.
[0143] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0144] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0145] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.
[0146] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0147] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.
[0150] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.
Claims
1. A data processing method, characterized in that, The method includes: When the first module transmits a data frame to the second module via a streaming connection, the latency parameter information of the data frame is obtained; wherein, the latency parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame; Based on the delay parameter information, the streaming performance analysis results are obtained, and the streaming performance analysis results are used for streaming function optimization. The step of obtaining the delay parameter information of the data frame includes: Obtain the outgoing data corresponding to the data frame; wherein the outgoing data has the encoded data frame; Identify the outgoing data and obtain the latency parameter information of the data frame; Specifically, a data format for recording outgoing data with encoded data frames is defined between the first module and the second module. The encoding start time, encoding end time, encoded data frame, and start transmission time information of the data frame are filled in on the first module side, while the peer reception time, peer decoding start time, and peer decoding completion time information of the data frame are filled in on the second module side. Furthermore, the second module is also used for parsing the outgoing data to obtain the latency parameter information of the data frame.
2. The method according to claim 1, characterized in that, The latency parameter information includes the encoding duration, transmission duration, and decoding duration of the data frame. The step of identifying the outgoing data and obtaining the latency parameter information of the data frame includes: Obtain the encoding start time and encoding end time corresponding to the data frame in the outgoing data; The encoding duration corresponding to the data frame is obtained based on the encoding start time and the encoding end time; Obtain the start transmission time and the receiving time of the data frame corresponding to the outgoing data; The transmission duration corresponding to the data frame is obtained based on the start transmission time and the receiving time of the peer end; Obtain the peer's start decoding time and peer's completion decoding time corresponding to the data frame in the outgoing data; The decoding duration corresponding to the data frame is obtained based on the peer's start decoding time and the peer's completion decoding time.
3. The method according to claim 1, characterized in that, The method further includes the step of generating outgoing data corresponding to the data frame. The generation of the outgoing data corresponding to the data frame includes: When the data frame is captured by the first module, a first storage area is requested to record the outgoing data corresponding to the data frame, and the time of requesting the first storage area is recorded in the encoding start timestamp field of the outgoing data to obtain the encoding start time corresponding to the data frame, and the encoding of the data frame begins. When the data frame is encoded, the time when the encoding of the data frame is completed is recorded in the encoding end timestamp field of the outgoing data to obtain the encoding end time corresponding to the data frame, and the encoded data frame is recorded in the encoded data field of the outgoing data. When an encoded data frame is sent from the first module to the second module, the time of sending the encoded data frame is recorded in the start transmission timestamp field of the outgoing data to obtain the start transmission time corresponding to the data frame. Upon receiving the encoded data frame sent by the first module, the time of receiving the encoded data frame is recorded in the peer-received timestamp field of the outgoing data to obtain the peer-received time corresponding to the data frame. When decoding the encoded data frame sent by the first module is started, the time when decoding of the encoded data frame begins is recorded in the peer-end start decoding timestamp field of the outgoing data to obtain the peer-end start decoding time corresponding to the data frame. When the encoded data frame is decoded, the time when the decoding of the encoded data frame is completed is recorded in the peer decoding completion timestamp field of the outgoing data to obtain the peer decoding completion time corresponding to the data frame.
4. The method according to claim 1, characterized in that, The latency parameter information also includes the total latency corresponding to the data frame; The total latency corresponding to the data frame is calculated based on the encoding duration, transmission duration, and decoding duration of the data frame.
5. The method according to claim 1, characterized in that, The method further includes: When the first module transmits data frames to the second module via a streaming connection, a first canvas is created in the desktop environment; The transmitted data frames are rendered and displayed using the first canvas.
6. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the delay parameter information of the data frame when the first module transmits the data frame to the second module through a streaming connection; wherein the delay parameter information includes at least one or more of the encoding duration, transmission duration, and decoding duration corresponding to the data frame. The second acquisition module is used to obtain streaming performance analysis results based on the delay parameter information, and the streaming performance analysis results are used for streaming function optimization. The first acquisition module is specifically used for: Obtain the outgoing data corresponding to the data frame; wherein the outgoing data has the encoded data frame; Identify the outgoing data and obtain the latency parameter information of the data frame; Specifically, a data format for recording outgoing data with encoded data frames is defined between the first module and the second module. The encoding start time, encoding end time, encoded data frame, and start transmission time information of the data frame are filled in on the first module side, while the peer reception time, peer decoding start time, and peer decoding completion time information of the data frame are filled in on the second module side. Furthermore, the second module is also used for parsing the outgoing data to obtain the latency parameter information of the data frame.
7. A head-mounted display device, characterized in that, The head-mounted display device includes: Memory is used to store executable computer instructions; A processor configured to execute the data processing method according to any one of claims 1-5, under the control of the executable computer instructions.
8. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, perform the data processing method according to any one of claims 1-5.