Motion vector generation method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a motion vector generation method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of display. The method comprises the steps that based on state information of a first scene, the value of an exchange variable pair of the first scene is determined, and the first scene is a scene where a target program runs currently; obtaining a value of a reserved variable of a second scene, wherein the second scene is a previous scene of the first scene; a forward motion vector for processing the first scene is generated based on the value of the swap variable pair of the first scene and the value of the retention variable of the second scene.

Description

Technical Field

[0001] The present application belongs to the field of display technology, and specifically relates to a motion vector generation method, device, electronic device, storage medium and program product. Background Art

[0002] In order to improve the realism of the display screen of electronic devices (such as game screen), the motion of objects in the game scene can be estimated to obtain motion vectors, and image processing such as temporal anti-aliasing (TAA) and motion blur effects can be performed based on the motion vectors.

[0003] Currently, existing methods for calculating motion vectors, such as block matching algorithms or optical flow methods, all perform motion estimation based on the image content displayed on the screen. These methods are only applicable to the motion estimation of translational objects. The accuracy of motion estimation of rotating and scaling objects in the screen is poor, and they are affected by lighting changes and object occlusion, resulting in low accuracy of the calculated motion vector. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a motion vector generation method, device, electronic device, storage medium and program product, which can improve the calculation accuracy of the motion vector and thus improve the display effect of the picture.

[0005] In a first aspect, an embodiment of the present application provides a motion vector generation method, the method comprising: determining the value of an exchange variable pair of a first scene based on status information of a first scene, the first scene being the scene in which a target program is currently running; obtaining the value of a retained variable of a second scene, the second scene being the previous scene of the first scene; generating a forward motion vector based on the value of the exchange variable pair of the first scene and the value of the retained variable of the second scene, the forward motion vector being used to process the first scene.

[0006] In a second aspect, an embodiment of the present application provides a motion vector generation device, which includes: a determination module, an acquisition module and an execution module. The determination module is used to determine the value of the exchange variable pair of the first scene based on the state information of the first scene, and the first scene is the scene currently running the target program. The acquisition module is used to obtain the value of the retained variable of the second scene, and the second scene is the previous scene of the first scene. The execution module is used to generate a forward motion vector based on the value of the exchange variable pair of the first scene determined by the determination module and the value of the retained variable of the second scene obtained by the acquisition module, and the forward motion vector is used to process the first scene.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, the value of the exchange variable pair of the first scene can be determined based on the state information of the first scene currently running in the target program, and then the value of the retained variable of the second scene can be obtained, and the second scene is the previous scene of the first scene, and then the forward motion vector for processing the first scene can be generated based on the value of the exchange variable pair of the first scene and the value of the retained variable of the second scene. In this scheme, the corresponding variable value can be obtained and processed based on the state information of the scene currently running by the program, that is, the data required to generate the forward motion vector can be obtained according to the program itself, so that the forward motion vector can be directly generated using these data, without the need to perform motion estimation based on the image content displayed on the screen, avoiding the error caused by analyzing the image content, and ensuring the accuracy of the motion vector. In this way, the calculation accuracy of the motion vector is improved, thereby improving the display effect of the screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is one of the flow charts of the motion vector generation method provided in the embodiment of the present application;

[0013] Figure 2 This is the second flowchart of the motion vector generation method provided in the embodiment of the present application;

[0014] Figure 3 This is the third flowchart of the motion vector generation method provided in the embodiment of the present application;

[0015] Figure 4 This is the fourth flowchart of the motion vector generation method provided in the embodiment of the present application;

[0016] Figure 5 This is the fifth flowchart of the motion vector generation method provided in the embodiment of the present application;

[0017] Figure 6 This is the sixth flowchart of the motion vector generation method provided in the embodiment of the present application;

[0018] Figure 7 is a schematic diagram of a motion vector generating device provided in an embodiment of the present application;

[0019] Figure 8 is a schematic diagram of the implementation process of the motion vector generation method provided in the embodiment of the present application;

[0020] Fig. 9 is a schematic diagram of a process of obtaining an exchange variable set and a reserve variable set provided in an embodiment of the present application;

[0021] Fig.10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0022] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] The terms "at least one (item)", "at least one of" and the like in this application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0026] The motion vector generation method, device, electronic device, storage medium and program product provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0027] Embodiments of the present application can be applied to scenarios where an electronic device calculates motion vectors for processing displayed images. Specifically, when an electronic device performs image processing such as TAA or motion blur on a displayed image, it is necessary to calculate motion vectors to implement the image processing scenario. For example, the scenario can be a variety of scenarios such as games, videos, VR or AR, and design visualization. Among them, TAA is an anti-aliasing technology for graphics rendering. Its main purpose is to reduce aliasing and other aliasing phenomena in graphics, thereby improving the visual quality and realism of the graphics. Motion blur is used to simulate the blurring effect of an object in motion. This effect is often used in movies, video games, and computer graphics to increase realism and dynamic effects.

[0028] For example, taking a game scene as an example, an electronic device needs to perform motion blur processing on the game screen of a racing game to enhance the authenticity of the game screen. The displayed content in the screen includes objects such as racing cars, tracks, dashboards, and operating controls. The electronic device needs to analyze the position change trend of the racing cars, tracks, and other environmental elements, calculate the motion vector of the current game screen, and thus perform motion blur processing on the racing game screen.

[0029] For example, in a video scene, the electronic device needs to perform motion blur processing on the room scene in the video to improve the authenticity of the picture. The display content in the picture includes objects such as sofas, tables, and computers. As the picture rotates, the electronic device needs to analyze the position change trend of objects such as sofas, tables, and computers, calculate the motion vector of the current picture, and then perform TAA processing on the room scene.

[0030] Motion vectors include forward motion vectors and backward motion vectors. Image processing technologies such as temporal anti-aliasing (TAA) and motion blur effects rely on backward motion vectors, which are calculated based on the pixel position change between the current frame and the previous frame. However, if these technologies are implemented only based on backward motion vectors, artifacts or ghosting may occur during image processing, and the accuracy of vector calculation will decrease when fast-moving objects or new objects appear in the scene, resulting in a decrease in image processing effect. It is necessary to calculate the forward motion vector to provide more accurate motion compensation information during motion blur and TAA processing, reduce artifacts or ghosting, and improve the accuracy of vector calculation.

[0031] Existing methods for calculating motion vectors include block matching algorithm, optical flow method, phase method and deep learning-based methods.

[0032] (1) The block matching algorithm can divide a frame in the video (called the reference frame) into blocks of fixed size. Then, the area that best matches these blocks is found in the next frame (called the current frame). The displacement vector generated by each block match is the motion vector, which indicates the new position of the corresponding block in the current frame. The block matching algorithm can be divided into: Full Search Algorithm, Three-Step Search Algorithm (TSS), Diamond Search Algorithm (DS), etc.

[0033] (2) The optical flow method can assume that the pixel intensity patterns of objects in the scene remain consistent between consecutive frames (i.e., the brightness constant assumption). By analyzing the changes in these patterns over time, the optical flow method can estimate the motion vector of each pixel between consecutive frames. Optical flow algorithms include the Lukas-Kanade algorithm and the Horn-Schunck method.

[0034] (3) The phase method can use the image frequency characteristics to analyze the motion between two frames in the Fourier transform domain. By calculating the cross power spectrum of the Fourier transform of the two frames and determining the peak, the global translation motion can be determined.

[0035] (4) Deep learning-based methods can use convolutional neural networks (CNN) and recurrent neural networks (RNN) to learn complex motion patterns from image sequences and generate motion vectors more accurately.

[0036] All of the above methods can realize the generation of forward motion vectors. However, when there are objects that are rotated, scaled, or occluded in the scene, or when there are changes in illumination, it is difficult for the block matching algorithm to find accurate matching blocks due to changes in the shape and size of the objects. The optical flow method cannot process pixels that change due to illumination, and occlusion will change the phase information of the image, reducing the calculation accuracy of the phase method. The deep learning-based method requires the use of neural networks to learn complex motion patterns, which has high requirements for hardware computing power. Its application is limited in scenarios such as real-time game rendering, which increases the computing cost and affects the smoothness of the game and user experience.

[0037] Existing methods for calculating motion vectors, such as block matching algorithms or optical flow methods, all perform motion estimation based on the image content displayed on the screen. These methods are only applicable to the motion estimation of objects that are translated in the screen. The accuracy of motion estimation of objects that are rotated and scaled in the screen is poor, and they are affected by lighting changes and object occlusion, resulting in low accuracy of the calculated motion vector.

[0038] The embodiments of the present application provide a motion vector generation method, device, electronic device, storage medium and program product, wherein the electronic device can obtain and process variable values ​​based on the status information of the scene in which the program is currently running, that is, the electronic device can obtain the data required to generate a forward motion vector according to the program itself, so that the forward motion vector can be directly generated using these data without the need to perform motion estimation based on the image content displayed on the screen, thereby avoiding errors caused by the electronic device analyzing the image content and ensuring the accuracy of the motion vector. In this way, the accuracy of the calculated motion vector is improved.

[0039] The execution subject of the motion vector generation method provided in the embodiment of the present application may be a motion vector generation device, and the motion vector generation device may be an electronic device, or a functional module or functional entity in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0040] Figure 1 A flow chart of a motion vector generation method provided by an embodiment of the present application is shown. Figure 1 As shown, the motion vector generation method provided in the embodiment of the present application may include the following steps 201 to 203.

[0041] Step 201: The electronic device determines the value of the exchange variable pair of the first scene based on the state information of the first scene.

[0042] In the embodiment of the present application, the first scenario mentioned above is the scenario in which the target program is currently running.

[0043] In the embodiment of the present application, the above-mentioned status information is used to indicate the first scene currently running.

[0044] It should be noted that the above-mentioned program is also called a shader program, which is a program entity that can be run independently in a graphics processing unit (GPU). It is a program entity created after the GPU compiles and links the shader code (i.e., the shader file described below), and can directly execute specific graphics or computing tasks on the GPU. For example, game programs, video programs, VR or AR programs, design programs, etc. Among them, each program can include at least one program, and each program is responsible for processing specific graphics effects or computing tasks, such as for rendering three-dimensional (3D) models, applying post-processing effects, performing physical simulations, etc.

[0045] It can be understood that a program is a program that can be run independently in the GPU. It is an entity created after the shader code is loaded into the GPU and compiled and linked. The target program can be determined by the target shader code. Similar underlying graphics API code can be: program = createProgram (shader code).

[0046] In an embodiment of the present application, the above target program can be used to calculate a motion vector, such as a backward motion vector.

[0047] It should be noted that the above motion vectors can be used to record the movement of objects in adjacent scenes in the scene, and to describe the pixel displacement of points in the picture between adjacent pictures. The motion vectors can include forward motion vectors and backward motion vectors.

[0048] The backward motion vector is the motion vector of an object in the current scene pointing to an object in the previous scene of the current scene, and is the data required for image processing such as TAA and motion blur. The forward motion vector is the motion vector of an object in the previous scene of the current scene pointing to an object in the current scene. Since there may be occlusion between the scenes, the forward motion vector cannot be equal to the opposite vector of the backward motion vector.

[0049] Optionally, in an embodiment of the present application, the above-mentioned state information may include relevant information of the currently running first scene, for example, the relevant information may include but is not limited to at least one of the following: scene information, variable information, and location information, etc.

[0050] In the embodiment of the present application, the above-mentioned exchange variable pair is used to exchange the scene data of the current scene and the previous scene of the current scene, that is, to exchange data between every two adjacent scenes rendered by the electronic device, including key information of the transition between the two scenes, such as motion vectors, illumination changes and other parameters. Each exchange variable pair may include two variable values, one variable value is used to indicate the scene data of the current scene, and the other variable value is used to indicate the scene data of the previous scene of the current scene.

[0051] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 2 As shown, the above step 201 can be implemented through the following step 201a.

[0052] Step 201a: The electronic device obtains the value of the exchange variable pair corresponding to the first scenario from the exchange variable pair set based on the state information of the first scenario.

[0053] In the embodiment of the present application, the value of the above-mentioned exchange variable pair includes a first variable value and a second variable value, the first variable value is used to indicate the scene data of the first scene, and the second variable value is used to indicate the scene data of the second scene.

[0054] In the embodiment of the present application, the second scene is the previous scene of the first scene.

[0055] In an embodiment of the present application, the above-mentioned exchange variable pair set may include multiple exchange variable pairs, each exchange variable pair has a corresponding function, such as saving texture information of a scene, lighting information of a scene or depth information of a scene.

[0056] In an embodiment of the present application, the electronic device can read the state information of the target program, and locate the value of the exchange variable pair from the exchange variable pair set according to the variable information included in the state information, through the variable index or variable name, for example, through the function program=createProgram(shader code).

[0057] In an embodiment of the present application, the electronic device can perform data exchange on the scene data of the first scene and the scene data of the second scene based on the exchange variable, that is, use the scene data of the second scene as the current scene data and use the scene data of the first scene as the previous scene data of the current scene data.

[0058] For example, the electronic device renders scene a and scene b, scene b is the next scene of scene a, the scene data of scene a is data c, and the scene data of scene b is data d. The electronic device can perform data exchange on the scene data of scene a and scene b, that is, use data d as the scene data of scene a, and use data c as the scene data of scene b.

[0059] In the embodiment of the present application, the electronic device can exchange the first variable value and the second variable value.

[0060] In the embodiment of the present application, after the variable values ​​are exchanged, the second variable value is used to indicate the scene data of the current scene, and the first variable value is used to indicate the scene data of the previous scene of the current scene.

[0061] It can be understood that the above-mentioned obtaining the value of the exchange variable pair corresponding to the first scenario from the exchange variable pair set is to obtain the first variable value and the second variable value after the variable values ​​are exchanged.

[0062] In this way, the electronic device can exchange the first variable value with the second variable value, change the first variable value to the second variable value, and change the second variable value to the first variable value, thereby realizing the exchange of scene data of the current first scene with scene data of the second scene.

[0063] Step 202: The electronic device obtains the value of the reserved variable of the second scene.

[0064] Optionally, in an embodiment of the present application, the value of the reserved variable of the second scenario is obtained according to the state information of the second scenario.

[0065] In an embodiment of the present application, the above-mentioned reserved variable set is used to retain the scene data of the previous scene of the current scene, that is, to retain the data of the previous scene rendered by the electronic device for use when the electronic device renders the current scene.

[0066] It should be noted that the above-mentioned exchange variable set and reserved variable set both include at least one variable value, wherein each variable value is used to represent different rendering data or calculation results when the electronic device performs image rendering. The main difference between them lies in the information content they carry and their use in the rendering process.

[0067] For example, one variable value might store the vertical motion vector from the previous frame, while another variable value might store the horizontal motion vector. These variable values ​​are passed from frame to frame to reflect the motion of objects in the scene.

[0068] In an embodiment of the present application, the electronic device can modify the retained variable value from scene data indicating the second scene to scene data indicating the first scene, that is, the electronic device can retain the scene data of the first scene as data of the second scene.

[0069] Step 203: The electronic device generates a forward motion vector based on the values ​​of the exchanged variable pairs of the first scene and the values ​​of the retained variables of the second scene.

[0070] In the embodiment of the present application, the forward motion vector is used to process the first scene.

[0071] It should be noted that, usually, in order to perform image processing such as TAA and motion blur, the electronic device can generate a backward motion vector based on the exchanged variable pairs and the retained variable values ​​before data exchange, that is, the electronic device can generate a backward motion vector based on the scene data of the current scene and the scene data of the previous scene of the current scene. In the embodiment of the present application, the electronic device can perform data exchange on the scene data of the first scene and the second scene by exchanging the variable pairs, and modify the retained variable values ​​from the scene data indicating the second scene to the scene data indicating the first scene, so as to exchange the scene data of the current scene and the scene data of the previous scene of the current scene, so that the electronic device can calculate the backward motion vector when the "previous scene of the current scene" is the first scene and the "current scene" is the second scene, that is, the forward motion vector when the "current scene" is the first scene and the "previous scene of the current scene" is the second scene.

[0072] It can be understood that (1) the target program provided by the game is used to calculate a high-precision backward motion vector. In the embodiment of the present application, the electronic device can use this target program as a black box to obtain a high-precision forward motion vector by mirroring the state.

[0073] (2) By implementing the process of the following step 302 through program automation, the values ​​of the exchange variable and the reserve variable can be selected.

[0074] (3) Track the call of the target program (DrawCall) while the game is running. While generating the backward motion vector, switch the conversion of the exchange variable and the reserve variable. Then call DrawCall in a black box manner to obtain a high-precision forward motion vector.

[0075] These three processes are all based on TAA or motion blur, using the target program used by TAA or motion blur to calculate the backward motion vector, and relying on the target program to generate high-precision forward motion vectors without destroying the game logic.

[0076] The embodiment of the present application provides a motion vector generation method, where the electronic device can obtain and process the corresponding variable value based on the state information of the scene currently running the program, that is, the electronic device can obtain the data required to generate the forward motion vector according to the program itself, so that the forward motion vector can be directly generated using these data, without the need to perform motion estimation based on the image content displayed on the screen, thus avoiding the error caused by the electronic device analyzing the image content and ensuring the accuracy of the motion vector. In this way, the calculation accuracy of the motion vector is improved, thereby improving the display effect of the screen.

[0077] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 3 As shown, before the above-mentioned step 201, the vector calculation method provided in the embodiment of the present application also includes the following steps 301 and 302.

[0078] Step 301: The electronic device obtains a target shader file corresponding to a target program.

[0079] In an embodiment of the present application, the above-mentioned shader file is a source code written in a shader language, which is used to compile into a shader program.

[0080] Optionally, in the embodiment of the present application, combined with Figure 3 ,like Figure 4 As shown, the above step 301 can be implemented through the following steps 301a to 301d.

[0081] Step 301a: The electronic device obtains a target texture identifier through an image processing task program.

[0082] In the embodiment of the present application, the above-mentioned target texture identifier is used to indicate the texture unit corresponding to the backward motion vector corresponding to the image processing task.

[0083] In an embodiment of the present application, the electronic device can determine the program that generates and uses the backward motion vector, that is, the program that performs the image processing task. The electronic device can then search for the uniform variable bound to the texture unit in the program, and the uniform variable is used to specify the texture unit that the texture should reference when sampling. The electronic device can then analyze the source code of the program or use a debugging tool to determine the texture unit bound to the uniform variable. Finally, the electronic device can query the texture ID bound to the texture unit from the application programming interface, that is, the above-mentioned target texture identifier.

[0084] Step 301b: The electronic device obtains a target frame buffer object (Frame Buffer Object, FBO) associated with the target texture identifier based on the target texture identifier.

[0085] It should be noted that when creating and configuring FBO, the program developer will bind a specific texture to the corresponding attachment of FBO. This information is usually stored in the device driver or graphics resource manager so that the associated FBO can be quickly found according to the texture identifier.

[0086] In the embodiment of the present application, the electronic device can find the target FBO according to the mapping relationship of the target texture identifier.

[0087] Step 301c: The electronic device locates the target program written into the target FBO through the frame buffer object binding function.

[0088] In the embodiment of the present application, when the FBO is bound, its information will be recorded, and then by querying the current binding status, the rendering operation directed to the FBO can be known. The electronic device can monitor and track the use of the FBO during the rendering process by querying the binding status, thereby locating the target program written to the target FBO.

[0089] Step 301d: Based on the target program, the electronic device obtains a target shader file corresponding to the target program from the mapping table.

[0090] In the embodiment of the present application, the above-mentioned mapping table includes at least one program and a shader file corresponding to each program.

[0091] It should be noted that when the program is initialized (for example, when a game or other program starts), the electronic device may compile multiple programs based on a predefined shader file, so that the electronic device can collect all created programs during the program initialization phase, and record the mapping relationship between each program and the corresponding shader file to form the above-mentioned mapping table.

[0092] In an embodiment of the present application, the electronic device can identify the identifier of the target program, and then use the identifier of the target program as a key value to query the mapping table to obtain the corresponding shader file.

[0093] Step 302: The electronic device parses the target shader file to obtain a set of exchange variable pairs.

[0094] In an embodiment of the present application, the electronic device can use the compiler front end glslang of the shader language to analyze the source code of the target shader file, and divide the input variables of the target shader file into an exchange variable set and a reserved variable set. Specifically, the electronic device can locate the output variable SV_Target0 of the shader, and then traverse the abstract syntax tree (Abstract SyntaxTree, AST) of the shader code through the graphics compiler interface, that is, the TIntermTraverser interface of glslang, to obtain a variable dependency directed acyclic graph with SV_Target0 as the root node and uniform variables and in variables as leaf nodes. Then the electronic device can locate two normalized device coordinates (Normalized DeviceCoordinates, NDC) variables, and use formula 1 to calculate the variable mv, formula 1 is as follows:

[0095] mv = pre_ndc * vec2(0.5) – cur_ndc Formula 1

[0096] Among them, the above mv is a variable used to characterize the motion vector of the object in the normalized device coordinate system, pre_ndc and cur_ndc are vectors in the NDC space, and the above vec2 is a two-dimensional floating-point vector in the shader language, which is used to indicate the position of a pixel or the coordinates of a point relative to the screen. Then the electronic device can take the variable mv as the center, trace back all upstream dependencies in the above variable dependency directed acyclic graph, and form two independent dependency sets: N-1 dependency set and N dependency set. The electronic device can compare the N-1 dependency set with the N dependency set, remove the overlapping parts (intersections) of the two dependency sets to eliminate redundant dependencies, and then the electronic device can filter out the exchange variable pair set according to the variable type and variable name. The remaining variables in the N dependency set except the exchange variable pair set are the retained variable set.

[0097] In this way, the electronic device can obtain the exchange variable pair set through the shader file of the target program, so that the variable values ​​can be exchanged.

[0098] Optionally, in the embodiment of the present application, combined with Figure 4 ,like Figure 5 As shown, before the above step 301a, the vector calculation method provided in the embodiment of the present application also includes the following step 401.

[0099] Step 401: The electronic device obtains a program of the image processing task from a mapping table based on a shader file of the image processing task.

[0100] In an embodiment of the present application, the electronic device can obtain a program of the image processing task corresponding to the shader file of the image processing task from a mapping table through a mapping relationship in the mapping table based on the shader file of the image processing task.

[0101] In this way, electronic devices can quickly locate the correct shader file through preset mapping relationships, reducing search and loading time, thereby speeding up the rendering process.

[0102] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 6 As shown, before the above step 201, the vector calculation method provided in the embodiment of the present application further includes the following step 501, and the above step 203 can be implemented by the following step 203a.

[0103] Step 501: The electronic device obtains the DrawCall function corresponding to the target program.

[0104] In the embodiment of the present application, the above-mentioned DrawCall function is a function corresponding to the interface of DrawCall.

[0105] It should be noted that the above DrawCall is a rendering call from the central processing unit (CPU) to the GPU, that is, the issuance of a rendering command. It is a key operation in the graphics rendering pipeline, which is used to pass the prepared vertex data, index data and related state settings (such as the shader program, texture, depth test, etc. used) to the GPU to complete the specific graphics drawing.

[0106] It can be understood that the above target program is used to calculate the backward motion vector, and the above target program calls the DrawCall function for processing when calculating the backward motion vector, that is, the above DrawCall function is used to calculate the backward motion vector.

[0107] Optionally, in an embodiment of the present application, the electronic device may monitor the program's call to the DrawCall function through a performance analysis tool, a graphics debugger, custom code, etc., thereby acquiring the DrawCall function.

[0108] It can be understood that the game program draws the screen content by using the CPU to set the program running state on the GPU, and then calls DrawCall to execute the program. The DrawCall interface is a few fixed functions, and tracking is to monitor the game program's calls to these fixed functions. The target program is created by the game program and generates backward motion vectors for TAA and motion blur algorithms. The purpose of tracking is to use the target program in a black box manner to generate forward motion vectors, that is, step 203a below.

[0109] Step 203a: The electronic device generates a forward motion vector through the DrawCall function based on the values ​​of the exchanged variable pair of the first scene and the values ​​of the retained variables of the second scene.

[0110] It should be noted that when the electronic device calls the DrawCall function to calculate the backward motion vector of the current scene, it is necessary to input the exchange variable values ​​and the reserved variable values ​​corresponding to the current scene and the previous scene of the current scene in the DrawCall function.

[0111] In an embodiment of the present application, the DrawCall function can calculate the backward motion vector of the game. The electronic device can exchange the values ​​in the exchange variable values ​​and set the retained variable values ​​to the variable values ​​corresponding to the scene data of the current scene, which is equivalent to a mirror conversion, so that the DrawCall function can generate a backward motion vector with the "current scene" as the second scene and the "previous scene of the current scene" as the first scene, that is, the forward motion vectors of the first scene and the second scene.

[0112] Exemplarily, the DrawCall function may be computeMV(exchange, reserve), where exchange is an exchange variable value and reserve is a reserved variable value. The electronic device may convert a function originally used to calculate a backward motion vector into a function to calculate a forward motion vector by fine-tuning parameters in the computeMV function (i.e., exchanging the exchange variable value and setting the reserved variable value).

[0113] In this way, electronic devices can obtain high-precision forward motion vectors by simply tracking key graphics rendering interface calls without modifying the logic and code of the program itself, reducing the consumption of computing resources and improving the accuracy of the motion vector.

[0114] It can be understood that the target program itself can generate a high-precision backward motion vector, and the forward motion vector can be calculated by exchanging the values ​​of the current exchange variable and the reserve variable used by the target program. The DrawCall of the target TAA is similar to a function: computeMV(exchange, reserve), which can calculate the backward motion vector of the game. As long as the exchange value is exchanged and the reserve value of the previous frame is set to the current frame, it is equivalent to a mirror conversion, and the forward motion vector of the game can be calculated.

[0115] In the embodiment of the present application, after calculating the forward motion vector, the electronic device may exchange the first variable value and the second variable value again, and modify the reserved variable value from the scene data indicating the first scene to the scene data indicating the second scene, so as to restore the scene data of the first scene and the second scene. In this way, the electronic device can restore the modified scene data to avoid affecting the rendering operation of the program itself.

[0116] The following is an exemplary description of the process of the motion vector generation method provided by the embodiment of the present application through a specific implementation method. Figure 7 As shown, the process may include steps 11 to 24 described below.

[0117] Step 11: (Preprocessing) Track the shader code corresponding to the program.

[0118] When the game starts, it will use shader code to create a program. When the program is initialized, it will get the shader code corresponding to the program and obtain a mapping table between the program (GPU program, used to run on the GPU) and the shader code (shader code).

[0119] Step 12: (Subsequently when the game is running) track TAA and motion blur program.

[0120] Use the TAA or motion blur shader code (obtained using the tool) to find the corresponding program (TAA program or motion blur program) from the output mapping table in step 11.

[0121] Step 13 obtains the backward motion vector texture ID (BK_MV).

[0122] Through the program corresponding to TAA or motion blur, the texture unit corresponding to the program backward motion vector is obtained, thereby obtaining the backward motion vector texture ID.

[0123] Step 14: Analyze the frame buffer object (FBO) written into BK_MV.

[0124] Find the target frame buffer object FBO according to the texture binding relationship when the frame buffer object (FBO) is created.

[0125] Step 15: Track the program of the frame buffer object (FBO).

[0126] Trace the frame buffer object binding function and locate the target program that writes to the target frame buffer object FBO.

[0127] Step 16: Record the program identifier and shader code.

[0128] According to the target program obtained in step 15 and the mapping table between the program and the shader code in step 11, the target shader code is found (this target shader code is used to generate the backward motion vector).

[0129] Step 17, glslang analyzes the shader code.

[0130] Use the shader language compiler front-end glslang to analyze the source code of the target shader code and divide the input variables of the target shader code into two categories: exchange variable set (the set contains variable pairs that can exchange values) and reserve variable set (the set contains some data reserved from the previous frame for use in the current frame). The scheme for dividing these two types of variable sets is as follows: Figure 8 shown.

[0131] like Figure 8As shown, first locate the output variable SV_Target0 of the shader, then use the TIntermTraverser interface provided by glslang to access the abstract syntax tree AST of the shader in visitor mode, and obtain the variable dependency directed acyclic graph with the SV_Target0 variable as the root node, then locate the two ndc variables, use these two ndc variables to search the variable dependency directed acyclic graph, obtain two sets, and then delete the intersection, and then get the exchange variable according to the variable type and name. After filtering out the exchange, the remaining variable in the N dependency set is the reserve variable.

[0132] The above are all preprocessing processes, which are ultimately used to determine the target shader code, exchange variable set, and reserve variable set corresponding to the game.

[0133] Step 18: Track the usage of the target program.

[0134] During the game running process, according to the determined target shader code, exchange variable set and reserve variable set corresponding to the game, the program being used is tracked. If it is the target program, the following steps are executed.

[0135] Step 19. Track the exchange and reserve variable settings.

[0136] By querying the current status of the target program, you can find the current values ​​of the exchange and reserve variables.

[0137] Step 20: Track the DrawCall of the target program.

[0138] DrawCall is an execution of the graphics rendering pipeline

[0139] Step 21: Exchange the exchange variable.

[0140] After DrawCall is executed, exchange the exchange variable value.

[0141] Step 22. Set the reserve variable.

[0142] Set the reserve variable value from the previous frame to the current frame.

[0143] Step 23: Call DrawCall.

[0144] Call the DrawCall function in step 20 again, which will generate the final game forward motion vector.

[0145] Step 24. Restore the values ​​of exchange and reserve settings.

[0146] In order not to affect the drawing status of the game itself, it is necessary to restore the status set in step 19.

[0147] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0148] The motion vector generation method provided in the embodiment of the present application can be executed by a motion vector generation device. In the embodiment of the present application, the motion vector generation method executed by the motion vector generation device is taken as an example to illustrate the motion vector generation device provided in the embodiment of the present application.

[0149] Fig. 9 A possible structural diagram of a motion vector generating device involved in some embodiments of the present application is shown. Fig. 9 As shown, the motion vector generating device 70 may include: a determining module 71 , an acquiring module 72 and an executing module 73 .

[0150] The determination module 71 is used to determine the value of the exchange variable pair of the first scene based on the state information of the first scene, where the first scene is the scene in which the target program is currently running.

[0151] The acquisition module 72 is used to acquire the value of the reserved variable of the second scene, where the second scene is the previous scene of the first scene.

[0152] The execution module 73 is used to generate a forward motion vector based on the values ​​of the exchange variable pair of the first scene determined by the determination module 71 and the values ​​of the retained variables of the second scene obtained by the acquisition module 72, and the forward motion vector is used to process the first scene.

[0153] In a possible implementation, the above-mentioned determination module 71 is specifically used to obtain the value of the exchange variable pair corresponding to the first scene from the exchange variable pair set; the value of the exchange variable pair includes a first variable value and a second variable value, the first variable value is used to indicate the scene data of the first scene, and the second variable value is used to indicate the scene data of the second scene.

[0154] In a possible implementation manner, the value of the reserved variable of the second scenario is obtained according to the state information of the second scenario.

[0155] In a possible implementation, the acquisition module 72 is further configured to acquire a target shader file corresponding to the target program before acquiring the value of the exchange variable pair corresponding to the first scene from the exchange variable pair set. The execution module 73 is further configured to parse the target shader file acquired by the acquisition module 72 to obtain the exchange variable pair set.

[0156] In a possible implementation, the acquisition module 72 is specifically used to: acquire a target texture identifier through a program of an image processing task, where the target texture identifier is used to indicate a texture unit corresponding to a backward motion vector corresponding to the image processing task; acquire a target frame buffer object FBO associated with the target texture identifier based on the target texture identifier; locate a target program written to the target FBO through a frame buffer object binding function; and acquire a target shader file corresponding to the target program from a mapping table based on the target program, where the mapping table includes at least one program and a shader file corresponding to each program.

[0157] In the embodiment of the present application, a motion vector generating device is provided, which can obtain and process the corresponding variable value based on the state information of the scene currently running the program, that is, the motion vector generating device can obtain the data required to generate the forward motion vector according to the program itself, so that the forward motion vector can be directly generated using these data without the need to perform motion estimation according to the image content displayed on the screen, thus avoiding the error caused by the motion vector generating device analyzing the image content and ensuring the accuracy of the motion vector. In this way, the calculation accuracy of the motion vector is improved, thereby improving the display effect of the screen.

[0158] The motion vector generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0159] The motion vector generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0160] The motion vector generation device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0161] Alternatively, if Fig.10 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a program or instruction that can be executed on the processor 1001, and when the program or instruction is executed by the processor 1001, the various steps of the above-mentioned motion vector generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0162] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0163] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0164] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0165] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0166] The processor 110 is used to determine the value of the exchange variable pair of the first scene based on the state information of the first scene, where the first scene is the scene in which the target program is currently running.

[0167] The processor 110 is further configured to obtain a value of a reserved variable of a second scene, where the second scene is a previous scene of the first scene.

[0168] The processor 110 is further configured to generate a forward motion vector based on the value of the exchange variable pair of the first scene and the value of the retained variable of the second scene acquired by the processor 110 , and the forward motion vector is used to process the first scene.

[0169] In one possible implementation, the processor 110 is specifically used to obtain the value of an exchange variable pair corresponding to a first scene from an exchange variable pair set; the value of the exchange variable pair includes a first variable value and a second variable value, the first variable value is used to indicate scene data of the first scene, and the second variable value is used to indicate scene data of the second scene.

[0170] In a possible implementation manner, the value of the reserved variable of the second scenario is obtained according to the state information of the second scenario.

[0171] In a possible implementation, the processor 110 is further configured to obtain a target shader file corresponding to the target program before obtaining the value of the exchange variable pair corresponding to the first scene from the exchange variable pair set. The processor 110 is further configured to parse the target shader file to obtain the exchange variable pair set.

[0172] In one possible implementation, the processor 110 is specifically used to: obtain a target texture identifier through a program of an image processing task, where the target texture identifier is used to indicate a texture unit corresponding to a backward motion vector corresponding to the image processing task; based on the target texture identifier, obtain a target frame buffer object FBO associated with the target texture identifier; locate a target program written to the target FBO through a frame buffer object binding function; based on the target program, obtain a target shader file corresponding to the target program from a mapping table, where the mapping table includes at least one program and a shader file corresponding to each program.

[0173] The embodiment of the present application provides an electronic device, which can obtain and process corresponding variable values ​​based on the state information of the scene currently running the program, that is, the electronic device can obtain the data required to generate the forward motion vector according to the program itself, so that the forward motion vector can be directly generated using these data, without the need to perform motion estimation based on the image content displayed on the screen, avoiding the error caused by the electronic device analyzing the image content, and ensuring the accuracy of the motion vector. In this way, the calculation accuracy of the motion vector is improved, thereby improving the display effect of the screen.

[0174] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and can achieve the same technical effect. To avoid repetition, it is not repeated here. The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, it is not repeated here.

[0175] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0176] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0177] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0178] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned motion vector generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0179] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0180] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned motion vector generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0181] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0182] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned motion vector generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0183] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0185] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for generating a motion vector, characterized in that: include: Determining, based on state information of a first scenario, a value of an exchange variable pair of the first scenario, the first scenario being a scenario in which the target program is currently running; Obtaining a value of a reserved variable of a second scene, where the second scene is a scene previous to the first scene; A forward motion vector is generated based on the values ​​of the swapped variable pairs of the first scene and the values ​​of the retained variables of the second scene, the forward motion vector being used to process the first scene.

2. The method according to claim 1, characterized in that The determining the value of the exchange variable pair of the first scenario includes: Acquire the value of the exchange variable pair corresponding to the first scenario from the exchange variable pair set; The value of the exchange variable pair includes a first variable value and a second variable value, the first variable value is used to indicate the scene data of the first scene, and the second variable value is used to indicate the scene data of the second scene.

3. The method according to claim 1, characterized in that The value of the reserved variable of the second scene is obtained according to the state information of the second scene.

4. The method according to claim 2, characterized in that: Before acquiring the value of the exchange variable pair corresponding to the first scenario from the exchange variable pair set, the method further includes: Obtain a target shader file corresponding to the target program; The target shader file is parsed to obtain the exchange variable pair set.

5. The method according to claim 4, characterized in that The obtaining of the shader file corresponding to the target program includes: Obtaining a target texture identifier through a program of an image processing task, wherein the target texture identifier is used to indicate a texture unit corresponding to a backward motion vector corresponding to the image processing task; Based on the target texture identifier, obtaining a target frame buffer object FBO associated with the target texture identifier; Locate the target program written into the target FBO through the frame buffer object binding function; Based on the target program, the target shader file corresponding to the target program is obtained from a mapping table, wherein the mapping table includes at least one program and a shader file corresponding to each program.

6. A motion vector generating device, characterized in that: include: Determine module, obtain module and execute module; The determination module is used to determine the value of the exchange variable pair of the first scenario based on the state information of the first scenario, where the first scenario is the scenario in which the target program is currently running; The acquisition module is used to acquire the value of the reserved variable of the second scene, where the second scene is the previous scene of the first scene; The execution module is used to generate a forward motion vector based on the value of the exchange variable pair of the first scene determined by the determination module and the value of the retained variable of the second scene acquired by the acquisition module, and the forward motion vector is used to process the first scene.

7. The device according to claim 6, characterized in that The determining module is specifically configured to obtain a value of an exchange variable pair corresponding to the first scenario from the exchange variable pair set; The value of the exchange variable pair includes a first variable value and a second variable value, the first variable value is used to indicate the scene data of the first scene, and the second variable value is used to indicate the scene data of the second scene.

8. The device according to claim 6, characterized in that The value of the reserved variable of the second scene is obtained according to the state information of the second scene.

9. The device according to claim 7, characterized in that The acquisition module is further configured to acquire a target shader file corresponding to the target program before acquiring the value of the exchange variable pair corresponding to the first scene from the exchange variable pair set; The execution module is further used to parse the target shader file obtained by the acquisition module to obtain the exchange variable pair set.

10. The device according to claim 9, characterized in that The acquisition module is specifically used for: Obtaining a target texture identifier through a program of an image processing task, wherein the target texture identifier is used to indicate a texture unit corresponding to a backward motion vector corresponding to the image processing task; Based on the target texture identifier, obtaining a target frame buffer object FBO associated with the target texture identifier; Locate the target program written into the target FBO through the frame buffer object binding function; Based on the target program, the target shader file corresponding to the target program is obtained from a mapping table, wherein the mapping table includes at least one program and a shader file corresponding to each program.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the motion vector generation method according to any one of claims 1 to 8 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the motion vector generation method according to any one of claims 1 to 8 are implemented.

Images