Frame insertion method and device, electronic equipment and storage medium

Abstract

The invention provides a frame insertion method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to the determination of texture rendering of a user interface UI, switching a rendering buffer from a first buffer to a second buffer, rendering the UI in the second buffer according to a target rendering instruction, and obtaining a first texture, obtaining a second texture which is obtained by rendering in the first buffer area and does not comprise the UI, carrying out frame insertion processing according to the second texture to obtain a frame insertion texture, fusing the frame insertion texture and the first texture to obtain a target frame insertion texture, and carrying out UI texture separation in the rendering process to obtain a first texture of the UI and the second texture which does not comprise the UI. And frame insertion is performed based on the second texture, so that the influence of the UI texture on frame insertion is avoided, and the accuracy of a frame insertion result is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to an interpolation method, apparatus, electronic device, and storage medium. Background Art

[0002] The screen refresh rates of current high-performance mobile devices have reached 120HZ, 144HZ, or even higher. However, a large number of mobile games still remain at 60HZ. By using interpolation technology, users can experience high-frame-rate games and enhance the gaming experience.

[0003] In related technologies, during the interpolation process, there are problems such as poor accuracy of the interpolation result and unsatisfactory image quality. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems in the related technologies to some extent.

[0005] To this end, this application proposes an interpolation method, apparatus, electronic device, and storage medium. By separating the UI texture during the rendering process to obtain the first texture of the UI and the second texture excluding the UI, interpolation based on the second texture avoids the influence of the UI texture on interpolation and improves the accuracy of the interpolation result.

[0006] An embodiment of one aspect of this application proposes an interpolation method, including:

[0007] In response to determining to perform texture rendering on the user interface UI, switch the rendering buffer from the first buffer to the second buffer;

[0008] Render the UI in the second buffer according to the target rendering instruction to obtain the first texture;

[0009] Obtain the second texture excluding the UI rendered in the first buffer;

[0010] Perform interpolation processing on the second texture to obtain an interpolated texture;

[0011] Fuse the interpolated texture and the first texture to obtain a target interpolated texture.

[0012] An embodiment of another aspect of this application proposes an interpolation apparatus, including:

[0013] A switching module, configured to switch the rendering buffer from the first buffer to the second buffer in response to determining to perform texture rendering on the user interface UI;

[0014] A rendering module, configured to render the UI in the second buffer according to the target rendering instruction to obtain the first texture;

[0015] An acquisition module, configured to acquire a second texture that does not include the UI and is rendered in the first buffer;

[0016] An interpolation module, configured to perform interpolation processing on the second texture to obtain an interpolated texture;

[0017] A fusion module, configured to fuse the interpolated texture and the first texture to obtain a target interpolated texture.

[0018] Another embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the foregoing aspect is implemented.

[0019] Another embodiment of this application provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the foregoing aspect is implemented.

[0020] Another embodiment of this application provides a computer program product, on which a computer program is stored. When the program is executed by a processor, the method described in the foregoing aspect is implemented.

[0021] The interpolation method, device, electronic device, and storage medium provided in this application, in response to determining the texture rendering of the user interface UI, switch the rendering buffer from the first buffer to the second buffer, render the UI in the second buffer according to the target rendering instruction to obtain the first texture, acquire the second texture that does not include the UI and is rendered in the first buffer, perform interpolation processing on the second texture to obtain the interpolated texture, fuse the interpolated texture and the first texture to obtain the target interpolated texture. By separating the UI texture during the rendering process to obtain the first texture of the UI and the second texture that does not include the UI, and performing interpolation based on the second texture avoids the influence of the UI texture on interpolation and improves the accuracy of the interpolation result.

[0022] Some of the additional aspects and advantages of this application will be given in the following description, some will become obvious from the following description, or be understood through the practice of this application. Description of the Drawings

[0023] The above-mentioned and / or additional aspects and advantages of this application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0024] Figure 1 It is a schematic flowchart of an interpolation method provided by an embodiment of this application;

[0025] Figure 2 It is a schematic flowchart of another interpolation method provided by an embodiment of this application;

[0026] Figure 3A Schematic diagram for identifying a target rendering instruction provided by an embodiment of the present application;

[0027] Figure 3B Schematic diagram for the interaction between an NPU and a GPU provided by an embodiment of the present application;

[0028] Figure 4 Flow schematic diagram of another frame interpolation method provided by an embodiment of the present application;

[0029] Figure 5 Architectural schematic diagram of a frame interpolation method provided by an embodiment of the present application;

[0030] Figure 6 Structural schematic diagram of a frame interpolation device provided by an embodiment of the present application;

[0031] Figure 7 Structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0032] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from start to end. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, but should not be construed as a limitation to the present application.

[0033] The frame interpolation method, device, electronic device, and storage medium of the embodiments of the present application will be described below with reference to the accompanying drawings.

[0034] Figure 1 Flow schematic diagram of a frame interpolation method provided by an embodiment of the present application.

[0035] In the embodiments of the present disclosure, it is exemplified that the frame interpolation method is configured in a frame interpolation device, and the frame interpolation device can be applied to any electronic device so that the electronic device can perform the frame interpolation function.

[0036] Among them, the electronic device can be any device with computing capabilities. For example, it can be a mobile terminal, and the mobile terminal can be a hardware device such as a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc. with various operating systems, touch screens, and / or display screens.

[0037] As Figure 1 shown, the method may include the following steps:

[0038] Step 101, in response to determining the texture rendering of the user interface UI, switch the rendering buffer from the first buffer to the second buffer.

[0039] The frame interpolation method of the embodiment of the present application can be applied to the field of video frame interpolation with user interaction, for example, the game field. In the game field, there is a layer of the texture of the user interface (UI). The layer of the UI texture refers to the interface where humans interact with the computer system (including software applications, websites, mobile applications, etc.), which is the part that users directly see and operate on.

[0040] Among them, the layer of the UI texture is not related to the movement of other objects. When performing frame interpolation, the UI image will cause great interference to the optical flow calculation or the calculation of the motion vector in the frame interpolation calculation process, resulting in poor accuracy of the frame interpolation result. During the process of drawing or rendering the display page, the painter's algorithm is usually used for texture rendering, that is, the scene farther away is rendered first, and then the scene closer is rendered to cover the farther part. In the game scene, the UI texture is the closest to the scene and is the last texture to be rendered. This makes it feasible to separate the UI texture. Only when other scenes have been rendered and the UI texture has not been rendered, the frame buffer is switched, that is, switched from the first buffer to the second buffer, and the UI texture is continuously rendered in the switched second frame buffer, so as to achieve the separation of the UI texture and the texture excluding the UI.

[0041] In the embodiment of the present application, it is necessary to determine the timing when the Graphics Processing Unit (GPU) performs the texture rendering of the UI. When it is determined that the GPU starts to perform the texture rendering of the UI, the buffer for the GPU to perform the texture rendering is switched from the first buffer to the second buffer. That is to say, the texture before the UI texture rendered by the GPU is in the first buffer. For the sake of easy distinction, the UI texture is called the first texture, and the texture excluding the UI is called the second texture. Furthermore, when it is determined that the GPU is going to render the first texture of the UI, the buffer is switched to enable the GPU to continue to render the first texture of the UI according to the target rendering instruction in the new buffer, and the different rendering results are saved in different frame buffers, and then the contents of these buffers are displayed on the screen. In this way, it can be ensured that the UI elements (such as menus, buttons, texts) are independently rendered from the game scene or other graphic contents of the application program, avoiding the influence of the first texture of the UI on the frame interpolation scene and improving the accuracy of the frame interpolation result.

[0042] It should be noted that the target rendering instruction refers to the rendering instruction used for rendering the first texture of the UI. For example, it is the rendering instruction of GLES or VulKan generated by the CPU based on the rendering request. Both the first buffer and the second buffer are buffers of the rendering pipeline, and can also be called the first frame buffer and the second frame buffer.

[0043] Step 102: Render the UI in the second buffer according to the target rendering instruction to obtain the first texture.

[0044] As an example, the target rendering instruction DrawCall is a rendering instruction initiated by the CPU to the GPU. For example, the glDrawElements function in OpenGL.

[0045] In the embodiment of the present application, the GPU renders in the second buffer according to the target rendering instruction to obtain the first texture of the UI, that is, the first texture is a texture including UI interface elements.

[0046] Step 103: Obtain the second texture that does not include the UI and is rendered in the first buffer.

[0047] The second texture is a texture including UI interface elements and is rendered by the GPU in the first buffer according to other rendering instructions. For example, it is the main application screen, such as the main application screen of a game scene.

[0048] Step 104: Perform frame interpolation processing on the second texture to obtain the interpolated texture.

[0049] The screen refresh rates of today's high-performance mobile devices have reached 120HZ, 144HZ, or even higher. However, a large number of mobile games still remain at 60HZ. Using frame interpolation technology, users can experience high-frame-rate games and enhance the gaming experience. However, during the frame interpolation process, the UI texture will interfere with the optical flow calculation, resulting in poor image quality when applied to mobile game frame interpolation. Therefore, in the embodiment of the present application, frame interpolation processing is performed based on the second texture that does not include the UI. Since the second texture does not include the UI, it will not affect the frame interpolation result during the frame interpolation process, improving the accuracy of the frame interpolation result.

[0050] Step 105: Blend the interpolated texture and the first texture to obtain the target interpolated texture.

[0051] In the embodiment of the present application, the interpolated texture does not include the UI, and the first texture includes the UI. Therefore, after the frame interpolation process is completed, the first texture and the second texture are blended to obtain the complete target interpolated texture to ensure the integrity of the finally displayed interpolated image. The target interpolated texture is inserted into the image buffer management component bufferQueue for display on the mobile device.

[0052] The target interpolated texture can be inserted as the texture after the second texture or as the texture before the second texture, which is not limited in this embodiment.

[0053] In the frame interpolation method according to the embodiments of the present application, in response to determining the texture rendering of the user interface UI, the rendering buffer is switched from the first buffer to the second buffer, the UI is rendered in the second buffer according to the target rendering instruction to obtain the first texture, the second texture that does not include the UI and is rendered in the first buffer is obtained, frame interpolation processing is performed according to the second texture to obtain the interpolated texture, the interpolated texture and the first texture are fused to obtain the target interpolated texture. By separating the UI texture during the rendering process, the first texture of the UI and the second texture that does not include the UI are obtained. Performing frame interpolation based on the second texture avoids the influence of the UI texture on frame interpolation and improves the accuracy of the frame interpolation result.

[0054] Based on the above embodiments, Figure 2 is a schematic flowchart of another frame interpolation method provided by the embodiments of the present application. As Figure 2 shown, the method includes the following steps:

[0055] Step 201, obtain the rendering request to be executed.

[0056] In the embodiments of the present application, the central processing unit (CPU) obtains the rendering request to be executed sent by the rendering engine and generates a corresponding rendering instruction based on the rendering request.

[0057] Step 202, send the rendering instruction to the GPU and query the identification information of the rendering instruction.

[0058] In the embodiments of the present application, the CPU sends the rendering instruction to the GPU, and the GPU uses the hook mechanism to capture the rendering instruction of the graphics application programming interface (OpenGL for Embedded Systems, GLES) issued by the game application, that is, intercepts the rendering instruction through the hook mechanism.

[0059] In an implementation manner of the embodiments of the present application, in the rasterization rendering pipeline, each rendering instruction drawcall corresponds to a rendering program program, and the core of the rendering program program is the shader file. Therefore, by identifying the shader file, the corresponding rendering program program can be identified, and thus the rendering instruction corresponding to the rendering program program can be identified. Among them, identifying the shader file can be implemented by using the shader compilation mechanism of the game. As an implementation manner, as Figure 3AAs shown, when the game download and installation are completed and it is launched for the first time, the shader files will be compiled. The shader files will be compiled and linked into shader programs, i.e., binary files. Then, the compiled binary files of the programs will be saved, and the own identification information will be appended to the end of the saved binary files. For example, it is the flag bits flags. Based on this identification information, when the game is launched or running, the program will be loaded. According to the identification information, it can be quickly identified whether the currently used program is the target program. Since each program has a corresponding rendering instruction, thus, according to the identification information, it can be identified whether the currently used program is the target program, and the rendering instruction corresponding to the target program is the target rendering instruction. The target rendering instruction is the rendering instruction used to render the UI texture. The target rendering instruction performs rendering by calling the corresponding rendering function. The rendering function is, for example, glDrawElement. Step 203, in response to determining that the rendering instruction is the target rendering instruction according to the identification information, query the state of the state machine of the Graphics Application Programming Interface GLES.

[0060] In the embodiments of the present application, according to the identification information, it can be determined that the rendering instruction is the target rendering instruction. The target rendering instruction can be used to render the UI texture, but the target rendering instruction does not only render the UI texture. The target rendering instruction performs rendering by calling the rendering function. However, during the execution of the target rendering instruction, the rendered objects are not necessarily the same. Therefore, the rendering behavior can be identified through the rendering instruction. The rendering behavior indicates which rendering pass the rendering has reached. If it is the target Render Pass, it is considered that the UI texture is to be rendered. Reaching the target Render Pass can also be understood as the number of times the gl*() function is called reaches the target number of times. For example, when the number of times the gl*() function is called reaches 12 times, the state of the state machine of GLES will change to the target state. If the CPU queries that the state of the state machine is the target state, it is confirmed that the frame buffer needs to be switched, that is, switched from the first buffer to the second buffer. Among them, the gl*() function is, for example, the glDrawElements function.

[0061] Step 204, in response to determining that the state of the state machine of GLES is the target state, determine that the GPU performs texture rendering on the UI.

[0062] Step 205, in response to determining that the Graphics Processing Unit GPU performs texture rendering on the User Interface UI, switch the rendering buffer from the first buffer to the second buffer.

[0063] In the embodiment of the present application, when it is determined that the rendering instruction is the target rendering instruction and the state of the state machine of GLES is the target state, it is determined that the GPU starts to perform texture rendering of the user interface (UI) based on the target rendering instruction, that is, it is determined that the switching time of the frame buffer arrives, and the UI texture is separately rendered by switching the frame buffer, realizing the separation of the UI texture.

[0064] Step 206, initialize and generate the initial texture of the UI in the second buffer.

[0065] In the embodiment of the present application, the initial texture of the UI includes the initial values of the color channels and the initial values of the transparency channels of each pixel point, and the initial texture is in RGBA format. As an implementation method, the GPU can first generate a texture in RGB format, and the RGB format texture includes the initial values of the color channels of each pixel point, and then add a transparency channel to the RGB format texture to convert it into an initial texture in RGBA format, so that the initial texture in RGBA format includes the initial values of the color channels and the initial values of the transparency channels of each pixel point. Among them, the color channels include three color channels: red (R), blue (B), and green (G); the transparency channel, usually called the Alpha channel, is used to represent the opacity of each pixel of the image in a digital image. The transparency value of the transparency channel ranges from 0 to 1, where 0 represents completely transparent; 1 represents completely opaque; and intermediate values represent different degrees of semi-transparency.

[0066] Step 207, according to the target rendering instruction, determine the target values of the color channels and the transparency channels of each pixel point in the initial texture to obtain the first texture of the UI.

[0067] In an implementation manner of the embodiment of the present application, the target rendering instruction includes all necessary information for rendering the UI texture. The texture of the UI is used to fill the elements of the UI. The GPU determines the first pixel points belonging to the elements of the UI and the second pixel points not belonging to the elements of the UI from the pixel points included in the initial texture according to the target rendering instruction. It can also be understood that the GPU can determine which pixel points in the initial texture are the pixel points to be rendered, called the first pixel points, and which pixel points are not the pixel points to be rendered, called the second pixel points, according to the information for rendering the UI texture included in the target rendering instruction.

[0068] Furthermore, the GPU updates the initial values of the color channels and the transparency channels of the first pixel points according to the first color value of the color channels and the first transparency value of the transparency channels of each first pixel point indicated by the target rendering instruction, to obtain the first color value of the color channels and the first transparency value of the transparency channels of the first pixel points. That is to say, after the update, the target value of the color channel of the first pixel point is the first color value and the target value of the transparency channel is the first transparency value.

[0069] Further, the initial values of the color channel and the transparency channel of the second pixel point are used as the second color value of the color channel and the second transparency value of the transparency channel of the second pixel point through the GPU, that is, the target value of the color channel of the second pixel point after determination is the second color value and the target value of the transparency channel is the second transparency value, so as to realize the determination of the target values of the color channels and the transparency channels of each pixel point to obtain the first texture of the UI.

[0070] As an example, the values of the color channel and the transparency channel of a pixel point can be expressed as (R, G, B, A), where A refers to the Alpha channel. For any pixel point k, the values of the color channel and the transparency channel of pixel point k in the initial texture are (0, 0, 0, 0). According to the target rendering instruction, if it is recognized that pixel point k is a pixel point to be rendered, the initial values of pixel point k are updated according to the values of the color channel and the transparency channel of pixel point k indicated by the target rendering instruction to obtain (r1_ui, g1_ui, b1_ui, 1), where the first transparency value is 1; if it is recognized that pixel point k is a pixel point not to be rendered, the initial values of pixel point k are updated according to the values of the color channel and the transparency channel of pixel point k indicated by the target rendering instruction to obtain (0, 0, 0, 0), where the second transparency value is 0. Similarly, the values of the color channels and the transparency channels of each pixel point can be determined to obtain the first texture of the UI.

[0071] Step 208: Obtain a second texture that does not include the UI and is rendered in the first buffer.

[0072] Step 209: Perform frame interpolation processing on the second texture to obtain an interpolated texture.

[0073] Step 210: Blend the interpolated texture and the first texture to obtain a target interpolated texture.

[0074] Among them, Step 208 and Step 210 can refer to the relevant explanations in the foregoing embodiments, and the principles are the same, so they will not be elaborated here.

[0075] In the frame interpolation method according to the embodiments of the present application, the rendering instructions can be identified by pre-determining the identification information corresponding to each rendering instruction. When it is determined that the rendering instruction is a target rendering instruction, it is determined whether it is the time to switch the frame buffer by identifying the state of the state machine. When it is determined that it is the switching time, the CPU switches the frame buffer, and then the GPU continues to perform rendering. During the rendering process, the RGB format is converted into the RGBA format to increase the transparency channel, and the value of the transparency channel is used for subsequent fusion of the first texture and the second texture to obtain a complete frame interpolation texture result, realizing the separation of the first texture and the second texture of the UI. Since the second texture does not include the interface elements of the UI, it is possible to avoid the interference of the UI layer on the frame interpolation result, and the accuracy of determining the frame interpolation result can be improved.

[0076] Based on the above embodiments, Figure 3B FIG. is a schematic diagram of the interaction between an NPU and a GPU provided by an embodiment of the present application, as Figure 3BAs shown in the figure, in the embodiment of the present application, the CPU sends rendering instructions to the GPU. The GPU uses the hook mechanism to capture the rendering instructions of the Graphics Application Programming Interface (OpenGL for Embedded Systems, GLES) issued by the game application, that is, intercepts the rendering instructions through the hook mechanism. In the embodiment of the present application, the rendering behavior of the rendering instructions can be identified according to the identification information of the intercepted rendering instructions, and the corresponding first rendering pass is determined according to the rendering behavior. When it is recognized that it is time to perform UI rendering, it is necessary to stop the relevant operations executed on the corresponding first rendering pass, that is, no longer continue to perform texture rendering in the original buffer area, that is, perform a rendering buffer switch, switch from the first buffer to the second buffer, and extract the texture from the first rendering pass to obtain a texture without UI, denoted as texN(i), that is, the second texture, realizing UI separation. Continue to render in the second buffer to obtain a newly created texture, which is a texture including UI, that is, the first texture, denoted as texUI, for fusing with the subsequent generated interpolated texture. The NPU completes feature extraction, that is, the optical flow feature map and the mask feature map, based on the rendered texture without UI, texN(i), and the cached texture without UI, texN(i - 1), and saves the calculation result in texUV. The NPU performs interpolation processing on the texture without UI, reducing the influence of UI on the optical flow information and improving the accuracy of determining the optical flow feature map. Then submit texUV to the GPU, and the GPU completes interpolation calculations such as sampling gridSample, realizing hardware acceleration of texture sampling through the GPU, and improving the overall execution efficiency of the model. The interpolation model is divided into two parts, one part is executed on the NPU, and the other part is executed on the GPU. The inference result of the model, that is, the interpolated texture, is texH. At this time, texH does not contain UI, which can avoid the interference of UI on the interpolation algorithm. Finally, it is necessary to fuse the texture texH and texUI to obtain the newly generated frame texMid, and insert texMid into the buffer manager bufferQueue for display on the mobile device.

[0077] Based on the above embodiments, Figure 4 is a schematic flowchart of another interpolation method provided by the embodiment of the present application, as Figure 4 shown, the method includes the following steps:

[0078] Step 401, in response to determining to perform texture rendering on the user interface UI, switch the rendering buffer from the first buffer to the second buffer.

[0079] Among them, performing texture rendering on the user interface UI is implemented by the Graphics Processing Unit (GPU).

[0080] Step 402: Render on the second buffer according to the target rendering instruction to obtain the first texture of the UI.

[0081] Step 403: Obtain the second texture that does not include the UI and is rendered in the first buffer.

[0082] Among them, steps 401 to 403 can refer to the relevant explanations in the foregoing embodiments. The principles are the same and will not be elaborated here.

[0083] Step 404: Obtain the historical texture that does not include the UI and is generated before the second texture is generated and sent by the GPU through the neural network processor NPU.

[0084] Among them, the historical texture is generated in the first buffer at a historical moment. The generation method of the second texture in the foregoing embodiments can be referred to and will not be elaborated here.

[0085] As an implementation manner, the historical texture is rendered by the GPU at a historical moment. The CPU reads the historical texture rendered in the GPU, sends the historical texture to the neural network processor (Neural Processing Unit, NPU), and stores it in the data structure stack.

[0086] As another implementation manner, the historical texture is stored in the storage unit of the GPU after being rendered by the GPU at a historical moment. After the second texture is generated, the CPU reads the second texture and the historical texture and sends the second texture and the historical texture to the NPU together.

[0087] As yet another implementation manner, the historical texture is rendered by the GPU at a historical moment. A shared memory is established between the GPU and the NPU. When the GPU renders and obtains the historical texture, it stores the historical texture in the shared memory, and the NPU reads the historical texture from the shared memory.

[0088] It should be noted that the method of obtaining the first texture in the NPU can refer to the method of obtaining the historical texture. The principles are the same and will not be elaborated here.

[0089] Step 405: Extract features from the second texture and the historical texture through the feature extraction module of the frame interpolation model running in the NPU to obtain an optical flow feature map and a mask feature map.

[0090] In the embodiments of the present application, the frame interpolation model depends on optical flow estimation. The frame interpolation model needs to use the Gridsample operator. The NPU has very poor support for this GridSample operator, and most of the time for running the frame interpolation model is wasted on this operator. GridSample is essentially a sampling function, and there is texture sampling acceleration hardware in the GPU, which can greatly accelerate sampling. Therefore, the frame interpolation model is split and deployed in the GPU and NPU respectively, that is, through hybrid programming of the GPU and NPU. The calculations of layers such as convolution, pooling, and activation are completed in the NPU, and the calculation of the GridSample is completed in the GPU, and the position of the GridSample operator is placed at the end of the frame interpolation model as much as possible. As shown in Table 1 below, Table 1 shows the structure of the adjusted frame interpolation model. Operators such as Conv convolution and pool pooling are used for feature extraction and are set to be performed in the NPU. The Gridsampel operator is located after operators such as Conv convolution and pool pooling and is set to be performed in the GPU. After adjusting the structure of the frame interpolation model, the first layer of the frame interpolation model is normally executed by the NPU, and the second and third layers are executed using the fragment shader of the GPU. Among them, the frame interpolation model is a neural network model.

[0091] Table 1

[0092]

[0093] Step 406, the frame interpolation module of the frame interpolation model running in the GPU performs frame interpolation processing according to the optical flow feature map and the mask feature map to obtain the interpolated texture.

[0094] Among them, the algorithm adopted by the sampling module is the Grid_sample operator, that is, the optical flow feature map is sampled through Grid_sample to obtain the sampling result.

[0095] As an implementation, the optical flow feature map is divided into a first optical flow feature map and a second optical flow feature map. The sampling result obtained by sampling using the first optical flow feature map is called the first sampling result, and the sampling result obtained by sampling using the second optical flow feature map is called the second sampling result. Among them, both the first optical flow feature map and the second optical flow feature map include the displacement information of pixel points in the horizontal direction and the vertical direction. The second texture is sampled using the first optical flow feature map to obtain the first sampling result, where the first sampling result indicates the starting position information of each pixel point in the second texture and the ending position information in the historical texture. The historical texture is sampled using the second optical flow feature map to obtain the second sampling result, where the second sampling result indicates the starting position information of each pixel point in the historical texture and the ending position information in the second texture. Furthermore, the target mask feature is weighted according to the first sampling result and the second sampling result to obtain the interpolated texture. As an implementation, the interpolated texture can be determined by the following formula:

[0096] image1 = warp_image0 * mask + warp_image2 * (1 - mask);

[0097] Among them, image1 is the interpolated texture, warp_image0 is the first sampling result, warp_image2 is the second sampling result, and mask is the mask feature.

[0098] Among them, the interpolation module includes a Mul dot product module, an Add addition module, and a Sub subtraction module. Weighting the mask feature map according to the sampling result of the interpolation module to determine the interpolated texture can make the generated interpolated texture more accurately reflect the motion information and visual consistency between adjacent frames. By using a hybrid programming of GPU and NPU, making full use of the sampling acceleration hardware unit of GPU and the convolution acceleration hardware unit of NPU, the time consumption of the entire path is reduced, and the time of overall model inference is reduced, so as to meet the real-time requirements of game interpolation. Among them, the optical flow information is mainly used to predict and simulate the pixel motion between adjacent frames during the interpolation process, and then generate new intermediate frames.

[0099] In the embodiments of the present application, in order to improve the efficiency of hybrid programming between NPU and GPU and reduce the overall inference time of the interpolation model, as an implementation, Figure 5 is the schematic architecture diagram of the interpolation method provided by the embodiments of the present application, as shown in Figure 5As shown in the figure, after the CPU determines that the timing for UI separation arrives, it switches the frame buffer from the first buffer to the second buffer, enabling the GPU to render in the second buffer to obtain the first texture of the UI. This process is non-blocking. Meanwhile, the NPU can perform feature extraction based on the second texture and the historical texture to obtain texture features. During the process of the feature extraction module of the frame interpolation model running in the NPU to extract features, a thread is created in the CPU to monitor whether the NPU has completed feature extraction. In the case of completing feature extraction, the CPU captures the state of the NPU, generates a notification message, and sends the notification message to the GPU, enabling the GPU to determine that the NPU feature extraction is completed. Meanwhile, the CPU configures the GPU based on the texture features, that is, sends the optical flow feature map and the mask feature map generated by the NPU to the gridsample module of the GPU's sampling module, enabling the GPU to use the sampling module (Gridsample) to sample the second texture and the historical texture according to the optical flow feature map to obtain the sampling result. Furthermore, the interpolation module of the frame interpolation model running in the GPU performs weighted processing on the mask feature map according to the sampling result to determine the interpolated texture. For specific reference to the foregoing explanations, it will not be elaborated here. By performing the sampling Gridsample operation in the GPU, the use of the sampling Gridsample operator is reduced, the read and write operations on the DDR memory are reduced, and the DDR bandwidth and power consumption can be saved. In addition, the output of the GPU is the result of frame interpolation, without holes, and there is no need to process the occlusion relationship again.

[0100] Step 407: Fuse the interpolated texture and the first texture to obtain the target interpolated texture.

[0101] In the embodiment of the present application, after the frame interpolation model is executed, for the obtained interpolated texture, it is necessary to add the UI texture, that is, based on the target value of the transparency channel in the first texture, fuse the interpolated texture and the first texture to ensure the integrity of the final display screen. Among them, the pixel points in the interpolated texture, the first texture, and the second texture are in one-to-one correspondence.

[0102] Among them, during the fusion process, for each pixel point of the first texture, the GPU queries the value of the transparency channel of the pixel point of the first texture. In one scenario, in response to determining by the GPU that the value of the transparency channel of the pixel point of the first texture is the first transparency value, update the value of the color channel of the pixel point in the interpolated texture with the first color value of the color channel of the pixel point of the first texture, and use the updated interpolated texture as the target interpolated texture.

[0103] In another scenario, in response to determining by the GPU that the value of the transparency channel of the pixel point of the first texture is the second transparency value, maintain the value of the color channel of the pixel point in the interpolated texture, and use the interpolated texture as the target interpolated texture.

[0104] As an example, for instance, the pixel values of each pixel point in the interpolated frame texture are represented as (ri_h, gi_h, bi_h), and the pixel values of each pixel point in the first texture are represented as (ri_ui, gi_ui, bi_ui, ai_ui), where i indicates which pixel it is. Among them, the interpolated frame texture is in RGB format, not RGBA format. Taking the first pixel point in the first texture as an example, that is, i = 1, query the value of the transparency channel of this pixel point. If the value of the transparency channel Alpha of this pixel point is 1, use the color channel values (r1_ui, g1_ui, b1_ui) of this pixel point in the first texture to replace the pixel value of this pixel point in the interpolated frame texture with (r1_ui, g1_ui, b1_ui); if the value of the transparency channel Alpha of this pixel point is 0, then do not change the pixel format of this pixel point in the interpolated frame texture, that is, the pixel value of this pixel point in the interpolated frame texture is (r1_h, g1_h, b1_h).

[0105] In the interpolated frame method of the embodiments of the present application, in the game field, there is also a layer of UI texture. This UI texture is not related to the movement of other objects, is fragmented and independent, is fixed, and is not in the same space. When calculating the optical flow / motion vector, the UI texture causes great interference. Before running the interpolated frame model, it is necessary to separate the UI texture to ensure that the input source of the interpolated frame model does not contain the UI texture, so as to reduce interference and reduce problems such as UI element drift, and ensure the correctness of the output result of the interpolated frame algorithm. After the interpolated frame model is executed, it is necessary to add the UI texture to ensure the integrity of the final display screen. By splitting the interpolated frame model and deploying it in the GPU and NPU respectively, that is, through GPU and NPU hybrid programming, the calculations of layers such as convolution, pooling, and activation are completed in the NPU, the calculation of GridSample is completed in the GPU, and the position of the gridSample operator is placed as close to the end of the interpolated frame model as possible to reduce the overall model inference time and meet the real-time requirements of game interpolation. In the process of fusing the interpolated frame texture and the first texture, fusion processing is performed based on the value of the transparency channel in the first texture to obtain the target interpolated frame texture, realizing the integrity of the final display screen.

[0106] To implement the above embodiments, the embodiments of the present application also propose an interpolated frame device.

[0107] Figure 6 It is a schematic structural diagram of an interpolated frame device provided by the embodiments of the present application.

[0108] As Figure 6 shown, the device may include:

[0109] A switching module 61, configured to switch the rendering buffer from a first buffer to a second buffer in response to determining to perform texture rendering on a user interface UI.

[0110] A rendering module 62, configured to render the UI in the second buffer according to a target rendering instruction to obtain a first texture.

[0111] An obtaining module 63, configured to obtain a second texture that does not include the UI and is rendered in the first buffer.

[0112] An interpolation module 64, configured to perform interpolation processing on the second texture to obtain an interpolated texture.

[0113] A fusion module 65, configured to fuse the interpolated texture and the first texture to obtain a target interpolated texture.

[0114] Furthermore, in an implementation manner of the embodiment of the present application, the apparatus further includes a determining module, configured to:

[0115] Obtain a rendering request to be executed; wherein the rendering request is used to generate a corresponding rendering instruction;

[0116] Send the rendering instruction to the GPU and query the identification information of the rendering instruction;

[0117] In response to determining that the rendering instruction is a target rendering instruction according to the identification information, query the state of the state machine of the graphics application programming interface GLES;

[0118] In response to determining that the state of the state machine of GLES is a target state, determine that the GPU performs texture rendering on the UI.

[0119] In an implementation manner of the embodiment of the present application, the rendering module 62 is further configured to:

[0120] Initialize and generate an initial texture of the UI in the second buffer; wherein the initial texture includes initial values of color channels and initial values of transparency channels of each pixel point;

[0121] According to the target rendering instruction, determine target values of color channels and target values of transparency channels of each pixel point in the initial texture to obtain the first texture of the UI.

[0122] In an implementation manner of the embodiment of the present application, the rendering module 62 is further configured to:

[0123] According to the target rendering instruction, determine first pixel points belonging to elements of the UI and second pixel points not belonging to elements of the UI from the pixel points included in the initial texture;

[0124] Updating the initial values of the color channel and the transparency channel of the first pixel point according to the first color value of the color channel and the first transparency value of the transparency channel of each first pixel point indicated by the target rendering instruction, to obtain the first color value of the color channel and the first transparency value of the transparency channel of the first pixel point; and,

[0125] Taking the initial values of the color channel and the transparency channel of the second pixel point as the second color value of the color channel and the second transparency value of the transparency channel of the second pixel point respectively, to obtain the first texture of the UI.

[0126] In an implementation manner of the embodiment of the present application, the frame interpolation module 64 is further configured to:

[0127] Obtaining, through a neural network processor NPU, a historical texture that does not include the UI and was generated before the generation of the second texture sent by the GPU;

[0128] Performing feature extraction on the second texture and the historical texture through a feature extraction module of a frame interpolation model running in the NPU, to obtain an optical flow feature map and a mask feature map;

[0129] Performing frame interpolation processing on the optical flow feature map and the mask feature map through a frame interpolation module of the frame interpolation model running in the GPU, to obtain the frame interpolation texture.

[0130] In an implementation manner of the embodiment of the present application, the fusion module 65 is further configured to:

[0131] For each pixel point of the first texture, querying the value of the transparency channel of the pixel point of the first texture;

[0132] In response to the value of the transparency channel of the pixel point being the first transparency value, updating the value of the color channel of the pixel point in the frame interpolation texture with the first color value of the color channel of the pixel point of the first texture;

[0133] Taking the updated frame interpolation texture as the target frame interpolation texture.

[0134] In an implementation manner of the embodiment of the present application, the fusion module 65 is further configured to:

[0135] In response to the value of the transparency channel of the pixel point being the second transparency value, maintaining the value of the color channel of the pixel point in the frame interpolation texture;

[0136] Taking the frame interpolation texture as the target frame interpolation texture.

[0137] It should be noted that the foregoing explanation of the method embodiment is also applicable to the device of this embodiment, and will not be elaborated here.

[0138] In the frame interpolation device according to the embodiment of the present application, in response to determining the texture rendering of the user interface UI, the rendering buffer is switched from the first buffer to the second buffer, the UI is rendered in the second buffer according to the target rendering instruction to obtain the first texture, the second texture that does not include the UI and is rendered in the first buffer is obtained, frame interpolation processing is performed according to the second texture to obtain the interpolated texture, the interpolated texture and the first texture are fused to obtain the target interpolated texture. By separating the UI texture during the rendering process, the first texture of the UI and the second texture that does not include the UI are obtained. Performing frame interpolation based on the second texture avoids the influence of the UI texture on frame interpolation and improves the accuracy of the frame interpolation result.

[0139] To implement the above embodiment, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the foregoing method embodiment is implemented.

[0140] To implement the above embodiment, the present application also proposes a non-transitory computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in the foregoing method embodiment is implemented.

[0141] To implement the above embodiment, the present application also proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the method described in the foregoing method embodiment is implemented.

[0142] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. For example, the electronic device 800 may be a mobile device, including a telephone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0143] Refer to Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0144] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0145] The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks, or optical disks.

[0146] The power component 806 provides power to various components of the electronic device 800. The power component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0147] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0148] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0149] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0150] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the electronic device 800. For example, the sensor component 814 can detect the on / off state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor component 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0151] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a wireless network based on communication standards, such as WiFi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0152] In an exemplary embodiment, the electronic device 800 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0153] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by a processor 820 of the electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0154] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0155] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0156] Any process or method description in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art of the embodiments of the present application.

[0157] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definable sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then storing it in a computer memory.

[0158] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0159] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0160] In addition, each functional unit in various embodiments of the present application may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0161] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. An interpolation method, characterized in that, Including: In response to determining to perform texture rendering on a user interface (UI), switch the rendering buffer from the first buffer to the second buffer; Render the UI in the second buffer according to a target rendering instruction to obtain a first texture; Obtain a second texture that does not include the UI and is rendered in the first buffer; Perform frame interpolation processing on the second texture to obtain an interpolated texture; Fuse the interpolated texture and the first texture to obtain a target interpolated texture.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a rendering request to be executed; wherein, the rendering request is used to generate a corresponding rendering instruction; Send the rendering instruction to a graphics processing unit (GPU) and query identification information of the rendering instruction; In response to determining that the rendering instruction is a target rendering instruction according to the identification information, query the state of a state machine of a graphics application programming interface (GLES); In response to determining that the state of the GLES state machine is a target state, determine that the GPU performs texture rendering on the UI.

3. The method according to claim 1, wherein The rendering the UI in the second buffer according to a target rendering instruction to obtain a first texture includes: Initialize and generate an initial texture of the UI in the second buffer; wherein, the initial texture includes initial values of color channels and initial values of transparency channels of each pixel point; According to the target rendering instruction, determine target values of color channels and target values of transparency channels of each pixel point in the initial texture to obtain the first texture of the UI.

4. The method according to claim 3, characterized in that, The determining target values of color channels and target values of transparency channels of each pixel point in the initial texture according to the target rendering instruction to obtain the first texture of the UI includes: According to the target rendering instruction, determine first pixel points belonging to elements of the UI and second pixel points not belonging to elements of the UI from the pixel points included in the initial texture; According to first color values of color channels and first transparency values of transparency channels of each first pixel point indicated by the target rendering instruction, update the initial values of color channels and initial values of transparency channels of the first pixel point to obtain the first color values of color channels and the first transparency values of transparency channels of the first pixel point; and Use the initial values of color channels and initial values of transparency channels of the second pixel point as second color values of color channels and second transparency values of transparency channels of the second pixel point respectively to obtain the first texture of the UI.

5. The method according to claim 1, wherein The performing frame interpolation processing on the second texture to obtain an interpolated texture includes: Obtain a historical texture that does not include the UI and is generated before the second texture is generated by a neural network processing unit (NPU) from the GPU; Extract features from the second texture and the historical texture through a feature extraction module of an interpolation model running in the NPU to obtain an optical flow feature map and a mask feature map; Perform frame interpolation processing according to the optical flow feature map and the mask feature map through an interpolation module of the interpolation model running in the GPU to obtain the interpolated texture.

6. The method according to claim 4, wherein Fusing the interpolated texture and the first texture to obtain a target interpolated texture includes: For each pixel of the first texture, query the value of the transparency channel of the pixel of the first texture; In response to the value of the transparency channel of the pixel being a first transparency value, update the value of the color channel of the pixel in the interpolated texture with the first color value of the color channel of the pixel of the first texture; Use the updated interpolated texture as the target interpolated texture.

7. The method according to claim 6, wherein The method further includes: In response to the value of the transparency channel of the pixel being a second transparency value, maintain the value of the color channel of the pixel in the interpolated texture; Use the interpolated texture as the target interpolated texture.

8. An interpolation frame device, characterized in that, It includes: A switching module, configured to switch the rendering buffer from a first buffer to a second buffer in response to determining to perform texture rendering on a user interface UI; A rendering module, configured to render the UI in the second buffer according to a target rendering instruction to obtain a first texture; An acquisition module, configured to acquire a second texture that does not include the UI and is rendered in the first buffer; An interpolation module, configured to perform interpolation processing on the second texture to obtain an interpolated texture; A fusion module, configured to fuse the interpolated texture and the first texture to obtain a target interpolated texture.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any one of claims 1-7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1-7 is implemented.

11. A computer program product, characterized in that, It includes a computer program which, when executed by the processor, implements the method described in any one of claims 1-7.

Images