Command stream repeated playback method and device of image frame

Abstract

The invention discloses a command stream repeated playback method and device for image frames. The method comprises the following steps: storing initial state data of any effective image frame generated by real-time rendering, and setting an initial state recovery command; and performing playback of any number of times on the command stream of any effective image frame generated by real-time rendering, and recovering the effective image frame to the initial state according to the initial state recovery command after each time of command stream playback is finished and before the next time of playback. According to the method and the device, command streams for drawing images such as OpenGLES and the like are freely and accurately controlled, so that the time for testing the optimization effect of the shader is saved to the greatest extent, the testing efficiency is improved, and the testing effect is indirectly improved.

Description

Technical Field

[0001] The invention belongs to the field of graphics drawing and automatic testing technology, and in particular relates to a method and device for repeatedly replaying a command stream of an image frame. Background Art

[0002] In the field of graphics rendering, it is still a big challenge to improve the rendering speed while ensuring the picture quality. In the rendering pipeline, how to use a given 3D model and material for rendering is defined by the shader. Specifically, the shader contains code and various parameter settings. The shader code needs to be written by technicians according to different shading models, and its complexity largely determines the picture quality and rendering speed. At present, in pursuit of high-quality pictures, the codes of shaders used in games, virtual reality and other applications are becoming more and more complex, so the calculation time of shaders is often the bottleneck in the rendering pipeline. For mobile terminals with relatively limited computing power, the burden brought by complex shaders is more obvious, which will not only make the frame rate lower and affect interactivity, but also increase the heating of the device and increase power consumption. Therefore, while ensuring the picture quality, it is a very important task to simplify the shader code to a certain extent and reduce the amount of calculation required by the shader.

[0003] Traditional shader simplification often relies on manual work by technicians, which requires high technical skills and experience. Furthermore, due to the high complexity of shaders, it takes a lot of time to manually explore the simplification space. The time not only includes the time required for manual code modification, but also the time required for testing the simplified shader.

[0004] The existing automatic simplification method of shaders mainly consists of three steps: first, automatic generation of simplified shaders, then measurement of the time consumption and error of simplified shaders, and finally screening based on the time consumption and error of simplified shaders. Among them, the measurement of the time consumption and error of simplified shaders requires modifying the original graphics program, and additionally implementing the function of replacing the target shader with the simplified shader and testing it in the same scenario. Therefore, the existing methods have the following two main disadvantages: First, most current games and virtual reality applications are implemented using third-party game engines. Since the drawing instruction stream that communicates with the graphics card is encapsulated in the underlying part of the game engine, and the general technical personnel who use the game engine are not familiar with or cannot access these underlying parts, it is not easy to accurately measure the time and error of simplifying the shader. The inability to accurately measure the simplified shader means that the optimal shader variant cannot be screened out in the end, and the simplification effect is unstable.

[0005] In addition, due to the reliance on the original graphics program, the existing methods need to repeatedly implement the measurement of shader time consumption and error for different graphics programs. To be widely applied to different graphics programs means a lot of repetitive work. Furthermore, the existing methods are not applicable to the case where the original graphics program code cannot be obtained. Therefore, the practicality of the existing methods is low.

[0006] Therefore, an automatic simplification method for shaders that can accurately measure the time and error of simplified shaders and does not rely on the original graphics program will have a better simplification effect and higher practicality. Evaluating the quality of shaders requires an environment that can control variables.

[0007] Existing tools have the functions of automatic optimization, automatic detection, and automatic iteration of shaders. The automatic detection stage requires replaying the OpenGLES command stream, replacing the optimized shader with the original command stream, and comparing the replay results to calculate the optimization effect. However, even if the time or number of times the command stream is used for the shader to be tested only occupies a small part of the entire replay process, the original command stream replay tool needs to replay the entire command stream once for each test. This makes the replay of other command streams that are not related to the shader to be tested occupy too much time in the entire effective test process, wasting a lot of time. Summary of the invention

[0008] In view of the above, the purpose of the present invention is to provide a method and device for repeated playback of the command stream of an image frame, to freely and accurately control the command stream of image drawing such as OpenGLES, to save the time used for testing the optimization effect of real-time rendering shaders to the greatest extent, to improve test efficiency, and indirectly improve test results.

[0009] To achieve the above-mentioned object of the invention, an embodiment provides a method for repeatedly replaying a command stream of an image frame, comprising the following steps: Save the initial state data of any valid image frame generated by real-time rendering and set the initial state recovery command; The command stream of any valid image frame generated by real-time rendering is replayed any number of times, and the valid image frame is restored to the initial state according to the initial state restoration command after each command stream replay ends and before the next replay.

[0010] Preferably, saving the initial state data of any valid image frame generated by real-time rendering and setting an initial state recovery command includes: Before the command stream is repeatedly played back, all commands of any valid image frame generated by real-time rendering are traversed to determine the resources that need to be saved. Specifically, the position of the current command stream pointer is saved, and all commands of the valid image frame are read in sequence through the current command stream pointer. When a command related to a resource is encountered, its parameters are read, and the parameters are converted into resources that need to be saved and recorded, and the position of the current command stream pointer is restored to the beginning; Save the resources that need to be saved and set the initial state recovery command corresponding to the resources, specifically: create a copy for the recorded resources that need to be saved, insert the initial state recovery command corresponding to the resources in the additional command stream for recovery, and record the copy into the parameters of the initial state recovery command; The resources include texture data and buffer data.

[0011] Preferably, saving the initial state data of any valid image frame generated by real-time rendering and setting an initial state recovery command includes: Before the command stream is repeatedly played back, all commands of any valid image frame are traversed to save various pipeline states of the current rendering and drawing program context. Specifically, a series of current pipeline states are queried using the interface of the current rendering and drawing program; for pipeline states that cannot be obtained using the interface query, an additional data structure is used to actively record the pipeline state; and the initial state recovery command corresponding to the pipeline state is inserted into an additional command stream for recovery.

[0012] Preferably, the command stream of any valid image frame generated by real-time rendering is played back any number of times, including: The position of the pointer pointing to the first command is saved before the command stream is repeatedly replayed, and the pointer of the command stream is restored to the saved position after the replay is completed and before the next replay starts.

[0013] Preferably, the method also includes: when replaying the command stream of the valid image frame for the first time, generating a recovery command for a specific command for the specific command in the command stream, and using the recovery command for the specific command to restore the valid image frame to its initial state during all subsequent replays of the command stream of the valid image frame.

[0014] Preferably, generating a recovery command of a specific command for a specific command in the command stream includes: For commands used to modify Uniform type variables in the pipeline, call the query command to query the initial variable data, and package the query data into the corresponding restore command; For commands to create textures, buffers, frame buffers, and semaphores, corresponding delete commands are generated as corresponding restore commands; For commands to delete textures, buffers, frame buffers, and semaphores, the corresponding create commands are generated as corresponding restore commands; For commands that modify the buffer format, call the query command to query the initial format of the buffer, and package the query data into the corresponding recovery command; For commands that map or unmap a buffer to memory, generate commands for the opposite operation and deduplicate them as corresponding restore commands; For commands related to defining pipeline vertex attributes, call query commands to query the initial vertex attribute related formats, and package the query data into corresponding recovery commands; When generating the recovery commands, repeated recovery commands are also pruned.

[0015] Preferably, the initial state recovery command and the recovery command of the specific command are both recorded through a bidirectional queue, and the initial state recovery command is recorded at the end of the queue, and the recovery command of the specific command is recorded at the head of the queue, so that the recovery operation can be restored in a stack manner or in a queue manner.

[0016] To achieve the above-mentioned purpose of the invention, the embodiment further provides a command stream repeated playback device for an image frame, comprising: A saving module, which is used to save the initial state data of any valid image frame generated by real-time rendering and set an initial state recovery command; The replay module is used to replay the command stream of any valid image frame generated by real-time rendering any number of times, and restore the valid image frame to the initial state according to the initial state recovery command after each command stream replay ends and before the next replay.

[0017] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the command stream repeated playback method of the above-mentioned image frame.

[0018] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the command stream of the above-mentioned image frame is repeatedly played back.

[0019] Compared with the prior art, the present invention has the following beneficial effects: In the present invention, the real-time rendering command stream is repeatedly replayed in units of image frames. Specifically, based on the initial state data of any valid image frame generated by real-time rendering and the setting of the initial state recovery command, the command stream of any valid image frame is replayed any number of times, and the valid image frame is restored to the initial state according to the initial state recovery command after each command stream replay ends and before the next replay. The redundant steps of the test are removed, the time of the test is saved, and the replay process becomes highly controllable and free. On this basis, corresponding recovery commands can also be constructed for specific commands in the command stream, and replay is performed based on the recovery command, which saves the time used for testing the optimization effect of the real-time rendering shader to the greatest extent, improves the test efficiency, and indirectly improves the test effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a command stream repeated playback method of an image frame provided by an embodiment; Figure 2 is a specific implementation flow chart of the command stream repeated playback method of the image frame provided in the embodiment; Figure 3 It is a structural schematic diagram of a command stream repeated playback device for image frames provided by an embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0023] The inventive concept of the present invention is: in order to achieve fast and accurate playback of rendering and drawing program command streams such as OpenGLES during shader detection, an embodiment provides a command stream repeated playback scheme for image frames. The scheme takes image frames as units and can freely and accurately control rendering and drawing command streams such as OpenGLES, thereby saving the time used for testing shader optimization effects to the greatest extent, improving test efficiency, and indirectly improving test results.

[0024] like Figure 1 As shown, the embodiment provides a method for repeatedly replaying a command stream of an image frame, comprising the following steps: S1, save the initial state data of any valid image frame generated by real-time rendering and set the initial state recovery command.

[0025] In the embodiment, before starting to replay the command stream in units of image frames, all commands of any image frame generated by real-time rendering are traversed, and it is determined whether there are commands in the image frame that cause the image frame to be unable to be replayed repeatedly. If not, the image frame is considered to be a valid image frame, and the initial state data of the valid image frame is saved and the initial state recovery command is set. The initial state data includes resources used by the image frame, and the resources include map data, buffer data, etc. The initial state data also includes the pipeline state of the rendering drawing program context such as OpenGLES, wherein the pipeline state includes the binding state of maps and buffers, the binding state of frame buffers, various state settings of the rasterization stage, the viewport and clipping window size, the depth test state, the template test and write state, etc.

[0026] For resource data, save the initial state data of any valid image frame generated by real-time rendering and set the initial state recovery command, including: Before the command stream is repeatedly played back, all commands of any valid image frame are traversed to determine the resources that need to be saved. Specifically, the position of the current command stream pointer is saved, and all commands of the valid image frame are read in sequence through the current command stream pointer. When a command related to a resource is encountered, its parameters are read, and the parameters are converted into resources that need to be saved and recorded, and the position of the current command stream pointer is restored to the beginning; Save the resources that need to be saved and set the initial state recovery command corresponding to the resources, specifically: create a copy for the recorded resources that need to be saved, insert the initial state recovery command corresponding to the resources in the additional command stream for recovery, and record the copy in the parameters of the initial state recovery command.

[0027] For pipeline status, save the initial status data of any valid image frame generated by real-time rendering and set the initial status recovery command, including: Before the command stream is repeatedly played back, all commands of any valid image frame are traversed to save various pipeline states of the current rendering and drawing program context. Specifically, a series of current pipeline states are queried using the interface of the current rendering and drawing program; for pipeline states that cannot be obtained using the interface query, an additional data structure is used to actively record the pipeline state; and the initial state recovery command corresponding to the pipeline state is inserted into an additional command stream for recovery.

[0028] S2, replaying the command stream of any valid image frame generated by real-time rendering any number of times, and restoring the valid image frame to the initial state according to the initial state restoration command after each command stream replay ends and before the next replay.

[0029] In the embodiment, the process of replaying the command stream of any valid image frame generated by real-time rendering any number of times is as follows: before the command stream is repeatedly replayed, the pointer position pointing to the first command is saved, and the pointer of the command stream is restored to the saved position after the replay is completed and before the next replay begins. Specifically, for the initial state data, the valid image frame is restored to the initial state by replaying the initial state restoration command after each command stream replay ends and before the next replay.

[0030] In an embodiment, the method, based on replaying based on the initial state recovery command, also includes: when replaying the command stream of the valid image frame for the first time, a recovery command of a specific command is also generated for the specific command in the command stream, and the recovery command of the specific command is used for recovery in all subsequent replays of the command stream of the valid image frame to restore the valid image frame to the initial state.

[0031] The specific command recovery command is generated for the specific command in the command stream, including: For commands used to modify Uniform type variables in the pipeline, call the query command to query the initial variable data, and package the query data into the corresponding restore command; For commands to create textures, buffers, frame buffers, and semaphores, corresponding delete commands are generated as corresponding restore commands; For commands to delete textures, buffers, frame buffers, and semaphores, the corresponding create commands are generated as corresponding restore commands; For commands that modify the buffer format, call the query command to query the initial format of the buffer, and package the query data into the corresponding recovery command; For commands that map or unmap a buffer to memory, generate commands for the opposite operation and deduplicate them as corresponding restore commands; For commands related to defining pipeline vertex attributes, call query commands to query the initial vertex attribute related formats, and package the query data into corresponding recovery commands; After generating each recovery command, repeated recovery commands are also trimmed to ensure recovery efficiency.

[0032] In the embodiment, the initial state recovery command and the recovery command of the specific command are both recorded through a bidirectional queue, and the initial state recovery command is recorded at the end of the queue, and the recovery command of the specific command is recorded at the head of the queue, so that the recovery operation can be restored in a stack manner or in a queue manner.

[0033] Based on the above command stream repeated playback method, the specific implementation process of the above method is also provided, such as Figure 2 As shown, including: 1. Before a certain image frame needs to be replayed, all commands contained in the image frame are traversed and the following preprocessing operations are performed: determine whether there are commands in the image frame that will cause the image frame to be unable to be replayed repeatedly, and record resource data such as textures and buffers whose content may change in the image frame.

[0034] 2. If it is determined that the current image frame can be repeatedly replayed, and the user chooses to repeatedly replay the current image frame, it will enter the continuous multiple replay process (referred to as the repeated frame process), otherwise it will be replayed normally without entering the repeated frame process.

[0035] 3. When entering the repeat frame process, first save the initial state data that can or must be saved before the frame replay starts and set the initial state recovery command to the two-way queue for recovery, as follows: (1) Create an identical copy of the resource data whose content may change for restoring the content of the resource in subsequent playback, and insert the initial recovery command corresponding to the resource into the end of the bidirectional queue for recovery; (2) Save the current pipeline state of each context and insert the initial state recovery command corresponding to the pipeline state into the tail of the bidirectional queue for recovery; (3) The activated map state is saved, and the initial state recovery command corresponding to the map state is inserted into the end of the bidirectional queue for recovery.

[0036] 4. Start replaying all commands of the current image frame. When replaying the image frame for the first time, generate corresponding recovery commands for specific commands, and insert these recovery commands into the head of the bidirectional queue for recovery.

[0037] 5. After replaying all commands of the current image frame, if you still need to continue replaying the current image frame, directly replay the bidirectional queue used for recovery to restore the current image frame to its appearance before replay, that is, the initial state.

[0038] 6. Continue to replay the current image frame. After the replay is completed, if you need to continue to play back, continue to replay the two-way queue used for recovery, and repeat this cycle.

[0039] 7. If the current image frame has been replayed, continue to replay the next image frame until all frames have been replayed and exit the program.

[0040] Based on the same inventive concept, Figure 3As shown, the embodiment also provides a device for repeated playback of the command stream of an image frame, including a saving module 31 and a replay module 32, wherein the saving module 31 is used to save the initial state data of any valid image frame generated by real-time rendering and set an initial state recovery command; the replay module 32 is used to replay the command stream of any valid image frame generated by real-time rendering any number of times, and restore the valid image frame to the initial state according to the initial state recovery command after each command stream replay ends and before the next replay.

[0041] It should be noted that the command stream replay device for image frames provided in the above embodiment should be illustrated by the division of the above functional modules when performing the command stream replay method. The above functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the command stream replay device for image frames provided in the above embodiment and the command stream replay method for image frames belong to the same concept. The specific implementation process is detailed in the command stream replay method for image frames, which will not be repeated here.

[0042] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the command stream repeated playback method for the above-mentioned image frame is implemented, specifically including the following steps: S1, saving the initial state data of any valid image frame generated by real-time rendering and setting an initial state recovery command; S2, replaying the command stream of any valid image frame generated by real-time rendering any number of times, and restoring the valid image frame to the initial state according to the initial state restoration command after each command stream replay ends and before the next replay.

[0043] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the command stream repeated playback method of the image frame described in S1-S2 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0044] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned command stream repeated playback method of the image frame is implemented, which specifically includes the following steps: S1, saving the initial state data of any valid image frame generated by real-time rendering and setting an initial state recovery command; S2, replaying the command stream of any valid image frame generated by real-time rendering any number of times, and restoring the valid image frame to the initial state according to the initial state restoration command after each command stream replay ends and before the next replay.

[0045] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0046] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for repeatedly replaying a command stream of an image frame, characterized in that: The following steps are involved: Save the initial state data of any valid image frame generated by real-time rendering and set the initial state recovery command; The command stream of any valid image frame generated by real-time rendering is replayed any number of times, and the valid image frame is restored to the initial state according to the initial state restoration command after each command stream replay ends and before the next replay.

2. The method for repeated playback of a command stream of an image frame according to claim 1, characterized in that: Save the initial state data of any valid image frame generated by real-time rendering and set the initial state recovery command, including: Before the command stream is repeatedly played back, all commands of any valid image frame generated by real-time rendering are traversed to determine the resources that need to be saved. Specifically, the position of the current command stream pointer is saved, and all commands of the valid image frame are read in sequence through the current command stream pointer. When a command related to a resource is encountered, its parameters are read, and the parameters are converted into resources that need to be saved and recorded, and the position of the current command stream pointer is restored to the beginning; Save the resources that need to be saved and set the initial state recovery command corresponding to the resources, specifically: create a copy for the recorded resources that need to be saved, insert the initial state recovery command corresponding to the resources in the additional command stream for recovery, and record the copy into the parameters of the initial state recovery command; The resources include texture data and buffer data.

3. The method for repeated playback of a command stream of an image frame according to claim 1, characterized in that: Save the initial state data of any valid image frame generated by real-time rendering and set the initial state recovery command, including: Before the command stream is repeatedly played back, all commands of any valid image frame generated by real-time rendering are traversed, and various pipeline states of the current rendering and drawing program context are saved. Specifically, a series of current pipeline states are queried using the interface of the current rendering and drawing program; for pipeline states that cannot be obtained using the interface query, an additional data structure is used to actively record the pipeline state; and the initial state recovery command corresponding to the pipeline state is inserted into an additional command stream for recovery.

4. The method for repeated playback of a command stream of an image frame according to claim 1, characterized in that: Replay any valid image frame command stream generated by real-time rendering any number of times, including: The position of the pointer pointing to the first command is saved before the command stream is repeatedly replayed, and the pointer of the command stream is restored to the saved position after the replay is completed and before the next replay starts.

5. The method for repeated playback of a command stream of an image frame according to claim 1, characterized in that: Also includes: When the command stream of the valid image frame is replayed for the first time, a recovery command for a specific command is generated for the specific command in the command stream, and the recovery command for the specific command is used to restore the valid image frame to its initial state during all subsequent replays of the command stream of the valid image frame.

6. The method for repeated playback of a command stream of an image frame according to claim 5, characterized in that: Generates recovery commands for specific commands in the command stream, including: For commands used to modify Uniform type variables in the pipeline, call the query command to query the initial variable data, and package the query data into the corresponding restore command; For commands to create textures, buffers, frame buffers, and semaphores, corresponding delete commands are generated as corresponding restore commands; For commands to delete textures, buffers, frame buffers, and semaphores, the corresponding create commands are generated as corresponding restore commands; For commands that modify the buffer format, call the query command to query the initial format of the buffer, and package the query data into the corresponding recovery command; For commands that map or unmap a buffer to memory, generate commands for the opposite operation and deduplicate them as corresponding restore commands; For commands related to defining pipeline vertex attributes, call query commands to query the initial vertex attribute related formats, and package the query data into corresponding recovery commands; When generating the recovery commands, repeated recovery commands are also pruned.

7. The method for repeated playback of a command stream of an image frame according to claim 1 or 5, characterized in that: The initial state recovery command and the recovery command of the specific command are both recorded through a bidirectional queue, and the initial state recovery command is recorded at the end of the queue, and the recovery command of the specific command is recorded at the head of the queue, so that the recovery operation can be restored in a stack manner or in a queue manner.

8. A command stream replay device for an image frame, characterized in that: include: A saving module, which is used to save the initial state data of any valid image frame generated by real-time rendering and set an initial state recovery command; The replay module is used to replay the command stream of any valid image frame generated by real-time rendering any number of times, and restore the valid image frame to the initial state according to the initial state recovery command after each command stream replay ends and before the next replay.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the command stream repeated playback method of the image frame described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the command stream repeated playback method of the image frame described in any one of claims 1-7 is implemented.

Images