Motion vector determination method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a motion vector determination method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first frame image and a second frame image which are continuous in time; a first drawing calling instruction corresponding to the first frame image and a second drawing calling instruction corresponding to the second frame image are determined, the first drawing calling instruction is a drawing calling instruction corresponding to a non-transparent object in the first frame image, and the second drawing calling instruction is a drawing calling instruction corresponding to a non-transparent object in the second frame image; the second drawing call instruction is a drawing call instruction corresponding to an opaque object in the second frame image; determining matched opaque objects in the first frame image and the second frame image based on the first drawing call instruction and the second drawing call instruction; and determining a motion vector between the first frame image and the second frame image based on the matched opaque objects in the first frame image and the second frame image. Through the method, the calculation precision of the motion vector is improved.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 63 / 585,756, filed with the U.S. Patent Office on September 27, 2023, and entitled “AMOTION VECTOR GENERATION FRAMEWORK FOR GAMES,” the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application belongs to the field of image processing technology, and specifically relates to a motion vector determination method, device, electronic device and storage medium. Background Art

[0004] The motion vector refers to the relative displacement between the current block (i.e., the current CU (Coding Unit)) and the similar block (i.e., the predicted block of the current CU). Motion vectors have been widely used in various fields. In the related inter-frame motion vector determination method, the calculation accuracy of the motion vector between two consecutive frames of images needs to be improved. Summary of the invention

[0005] In view of the above problems, the present application proposes a motion vector determination method, device, electronic device and storage medium to improve the above problems.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a motion vector, the method comprising: acquiring a first frame image and a second frame image that are continuous in time; determining a first draw call instruction corresponding to the first frame image, and a second draw call instruction corresponding to the second frame image, wherein the first draw call instruction is a draw call instruction corresponding to an opaque object in the first frame image, and the second draw call instruction is a draw call instruction corresponding to an opaque object in the second frame image; based on the first draw call instruction and the second draw call instruction, determining matching opaque objects in the first frame image and the second frame image; based on the matching opaque objects in the first frame image and the second frame image, determining a motion vector between the first frame image and the second frame image.

[0007] In a second aspect, an embodiment of the present application provides a motion vector determination device, the device comprising: an image acquisition unit, configured to acquire a first frame image and a second frame image that are continuous in time; an instruction determination unit, configured to determine a first drawing call instruction corresponding to the first frame image, and a second drawing call instruction corresponding to the second frame image, wherein the first drawing call instruction is a drawing call instruction corresponding to an opaque object in the first frame image, and the second drawing call instruction is a drawing call instruction corresponding to an opaque object in the second frame image; an object determination unit, configured to determine matching opaque objects in the first frame image and the second frame image based on the first drawing call instruction and the second drawing call instruction; and a motion vector determination unit, configured to determine a motion vector between the first frame image and the second frame image based on matching opaque objects in the first frame image and the second frame image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, wherein the above method is executed when the program code is run.

[0010] The embodiments of the present application provide a method, device, electronic device and storage medium for determining a motion vector. First, a first frame image and a second frame image that are continuous in time are obtained, and a first drawing call instruction corresponding to the first frame image and a second drawing call instruction corresponding to the second frame image are determined, wherein the first drawing call instruction is a drawing call instruction corresponding to an opaque object in the first frame image, and the second drawing call instruction is a drawing call instruction corresponding to an opaque object in the second frame image, and then based on the first drawing call instruction and the second drawing call instruction, matching opaque objects in the first frame image and the second frame image are determined, and finally based on the matching opaque objects in the first frame image and the second frame image, a motion vector between the first frame image and the second frame image is determined. Through the above method, matching opaque objects in two frames of images are determined based on the drawing call instruction corresponding to each frame image, so that the determination accuracy of opaque objects can be improved, and on the basis of improving the determination accuracy of opaque objects, the calculation accuracy of motion vectors between two frames of images can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram showing an application scenario of the motion vector determination method proposed in an embodiment of the present application;

[0013] Figure 2 A flow chart of a method for determining a motion vector proposed in an embodiment of the present application is shown;

[0014] Figure 3 A logical schematic diagram of instruction flow transmission in an image rendering process in an embodiment of the present application is shown;

[0015] Figure 4 A flow chart of a method for determining a motion vector proposed in another embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram showing a method of calculating the similarity between a first matching feature and a second matching feature in another embodiment of the present application is shown;

[0017] Figure 6 A flow chart of a method for determining a motion vector proposed in another embodiment of the present application is shown;

[0018] Figure 7 A flow chart of a method for determining a motion vector proposed in another embodiment of the present application is shown;

[0019] Figure 8 A structural block diagram of a motion vector determination device proposed in an embodiment of the present application is shown;

[0020] Fig. 9 A structural block diagram of a motion vector determination device proposed in an embodiment of the present application is shown;

[0021] Fig.10 A structural block diagram of an electronic device for executing a motion vector determination method according to an embodiment of the present application is shown;

[0022] Fig.11 A storage unit in an embodiment of the present application is shown for storing or carrying a program code for implementing a motion vector determination method according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] Motion vectors are widely used in various fields, such as games or images. In computer graphics, especially in rendering engines, the motion vectors of 3D objects are implemented through rendering channels. Although the graphics engines Unity API and AMD Cauldron support motion vector rendering channels, and the algorithms implemented in the shader are known, rendering only applies to the same objects given between frames. Even in Unity, if the application uses static batching, it first renders a full-screen frame as a post-processing rendering channel to calculate motion vectors, but it is still necessary to handle dynamic objects. In order to find the same object between frames, Unity uses a method called "object ID". During the rendering process, Unity assigns a unique identifier (ID) to each object or material in the scene. During the motion vector pass, these IDs are stored in a buffer along with the motion vectors. In subsequent frames, Unity uses this information to match objects based on IDs and determine how they move from one frame to another. By using this object ID information and motion vector data, Unity reconstructs the screen space motion of the object.

[0025] However, in most games / graphics applications, APIs or IDs related to motion vectors are not fully exposed. When motion vectors are needed for prototyping or product execution, developers have no direct method or API available, which results in the accuracy of the calculated motion vectors being left to be improved.

[0026] Therefore, the inventor proposes a motion vector determination method, device, electronic device and storage medium in the present application. First, a first frame image and a second frame image that are continuous in time are obtained, and a first drawing call instruction corresponding to the first frame image and a second drawing call instruction corresponding to the second frame image are determined, wherein the first drawing call instruction is a drawing call instruction corresponding to an opaque object in the first frame image, and the second drawing call instruction is a drawing call instruction corresponding to an opaque object in the second frame image, and then based on the first drawing call instruction and the second drawing call instruction, matching opaque objects in the first frame image and the second frame image are determined, and finally based on the matching opaque objects in the first frame image and the second frame image, a motion vector between the first frame image and the second frame image is determined. Through the above method, matching opaque objects in two frames of images are determined based on the drawing call instruction corresponding to each frame image, so that the determination accuracy of opaque objects can be improved, and on the basis of improving the determination accuracy of opaque objects, the calculation accuracy of motion vectors between two frames of images can be further improved.

[0027] The following is an introduction to the application environment of the motion vector determination method provided in the embodiment of the present application:

[0028] See also Figure 1 The motion vector method provided in the embodiment of the present application can be applied to a motion vector framework architecture 100, in which a frame analysis tool 110 and a motion vector calculation framework 120 may be included.

[0029] Among them, the frame analysis tool (Frame Profiler) 110 is a graphics debugging tool that supports capturing multiple frames into a single file. Specifically, by capturing a single frame based on Vulkan or OpenGL ES or continuous frames of OpenGLES applications, the graphics standard API protocol execution sequence at different stages of the rendering and computing pipeline is analyzed. The call relationship is decomposed layer by layer, so that the debugging user can clearly understand the input and output resource relationship, so as to identify, locate and solve the rendering and computing problems of the application.

[0030] The motion vector calculation framework 120 may include a pre-processing module 121 and a motion vector calculation module 122. The pre-processing module 121 may include a frame parser and a draw call cache; the motion vector calculation module 122 may include a draw call matching and a motion vector rendering pass.

[0031] The frame parser is used to parse each frame from a single capture file; the draw call cache is used to cache the motion vector calculation functions (OpenGL functions) required for draw calls and call stacks; the draw call matching is used to filter transparent objects and match opaque objects with rendering draw calls; the motion vector rendering pass is used to create and implement the motion vector rendering pass and write the value to the frame buffer.

[0032] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0033] See also Figure 2 , an embodiment of the present application provides a method for determining a motion vector, the method comprising:

[0034] Step S110: Acquire a first frame image and a second frame image that are consecutive in time.

[0035] In an embodiment of the present application, the first frame image and the second frame image are two frame images acquired continuously in time. As one way, the first frame image and the second frame image can be two frame images acquired continuously in time after the game application is run. As another way, the first frame image and the second frame image can be two frame images acquired continuously in time from a video stream. Of course, the first frame image and the second frame image can also be two frame images acquired continuously in time from other places, which is not specifically limited here. Among them, continuous in time refers to continuous in sampling time; sampling time refers to a preset sampling frequency.

[0036] As a way, in the case where the first frame image and the second frame image are two frame images that are continuous in time after the game application is run. When it is detected that the game application is in a running state, in response to the image acquisition instruction, the first frame image and the second frame image are captured by the frame analysis tool (FrameProfiler). Among them, whether the game application is in a running state can be determined by detecting the state identifier of the game application. Specifically, each game application can correspond to multiple state identifiers, each state identifier is used to characterize that the game application is in a different state, wherein the state of the game application may include a startup state, a running state, and a dormant state, etc., which are not specifically limited here. In an embodiment of the present application, different states correspond to different state identifiers, for example, the state identifier of the startup state can be represented by "QD", the state identifier of the running state can be represented by "YX", and the state identifier of the dormant state can be represented by "XM". In this case, when the state identifier of the game application is detected as "YX", it can be determined that the game application is in a running state. At this time, in response to the image acquisition instruction, the first frame image and the second frame image that are continuous in time can be captured by the frame analysis tool.

[0037] As another way, the first frame image and the second frame image may be two frame images that are continuous in time obtained from a video stream. After the video stream is acquired, in response to the image acquisition instruction, two frame images that are continuous in time are arbitrarily selected from the video stream through the frame analysis tool (FrameProfiler). Among them, the video stream may be composed of multiple frame images. The electronic device can play the frame images quickly and sequentially, so that the user can see the dynamic picture composed of the continuously played frame images through the display of the electronic device. The number of frame images played by the electronic device in a unit time can be identified by the frame rate. The higher the frame rate, the more frame images the electronic device plays in a unit time, and the corresponding dynamic picture is clearer and more realistic.

[0038] In the above two cases, the image acquisition instruction may be an instruction triggered by a preset control in the frame analysis tool, or may be an instruction sent by other devices with which the electronic device has established a communication connection, and no specific limitation is made here.

[0039] Optionally, in the embodiment of the present application, the first frame image and the second frame image are captured frame images that have been displayed on a display of an electronic device.

[0040] Step S120: determining a first draw call instruction corresponding to the first frame image, and a second draw call instruction corresponding to the second frame image, wherein the first draw call instruction is a draw call instruction corresponding to an opaque object in the first frame image, and the second draw call instruction is a draw call instruction corresponding to an opaque object in the second frame image.

[0041] In the embodiment of the present application, the first drawing call instruction is a determined instruction for rendering an opaque object in the first frame image; the second drawing call instruction is a determined instruction for rendering an opaque object in the second frame image. Among them, one opaque object corresponds to one first drawing call instruction, so the number of first drawing call instructions corresponding to the first frame image in the embodiment of the present application can be determined by the number of opaque objects included in the first frame image; similarly, the number of second drawing call instructions corresponding to the second frame image can be determined by the number of opaque objects included in the second frame image. Among them, it can be known that the motion vector is only applied to opaque objects, so the opaque object in the embodiment of the present application refers to an object that can be used to calculate the motion vector between two frames of images.

[0042] It is understandable that, for each frame image, the application can display the electronic device to perform corresponding drawing through the rendering instruction stream, thereby obtaining display information and displaying it through the display.

[0043] Exemplary, combined Figure 3 , when a frame image needs to be displayed, the application can issue a rendering instruction stream to instruct the electronic device to perform rendering operations according to the rendering instruction stream. The central processing unit (CPU) of the electronic device can receive the rendering instruction stream and call the corresponding application programming interface (API) in the rendering environment installed in the electronic device according to the rendering instruction stream. The CPU can instruct the graphics processing unit (GPU) with image rendering function in the electronic device to perform corresponding rendering operations by calling the API. The rendering result obtained after the GPU performs a series of rendering operations can correspond to the display information. In some implementations, the display information may include color information, brightness information, depth data, normal information, etc. corresponding to each pixel point in the current frame image. When the frame image needs to be displayed, the display can obtain the rendering result and display it accordingly.

[0044] That is to say, when a plurality of frame images need to be displayed quickly, the electronic device needs to perform corresponding rendering operations according to each rendering instruction stream issued by the application program in order to obtain display information of the corresponding frame images.

[0045] When instructing an electronic device to render a frame of image, an application may issue an instruction stream including multiple draw call instructions (Drawcall). In an embodiment of the present application, the first draw call instruction corresponding to the first frame of image may be at least one draw call instruction determined from the instruction stream corresponding to the first frame of image for rendering an opaque object included in the first frame of image; the second draw call instruction corresponding to the second frame of image may be at least one draw call instruction determined from the instruction stream corresponding to the second frame of image for rendering an opaque object included in the second frame of image.

[0046] It should be noted that, when a frame image includes multiple opaque objects that need to be drawn separately, the application program will issue Drawcalls corresponding to the multiple opaque objects respectively. In this way, the electronic device can perform a rendering process for each opaque object.

[0047] In the embodiment of the present application, the instruction streams corresponding to the first frame image and the second frame image may be stored in a preset buffer area. When it is necessary to determine the first drawing call instruction corresponding to the first frame image and the second drawing call instruction corresponding to the second frame image, they may be obtained from the instruction streams corresponding to the first frame image and the second frame image respectively stored in the preset buffer area.

[0048] Step S130: determining matching opaque objects in the first frame image and the second frame image based on the first drawing call instruction and the second drawing call instruction.

[0049] In the embodiment of the present application, after obtaining the first draw call instruction corresponding to the first frame image and the second draw call instruction corresponding to the second frame image, the matching opaque objects in the first frame image and the second frame image can be determined by a draw call matching algorithm. The matching opaque objects in the first frame image and the second frame image refer to opaque objects that exist in the first frame image and the second frame image and whose similarity is greater than a preset similarity. In the embodiment of the present application, the draw call matching algorithm is an algorithm pre-set to determine matching opaque objects in the first frame image and the second frame image.

[0050] Step S140: determining a motion vector between the first frame image and the second frame image based on matching opaque objects in the first frame image and the second frame image.

[0051] In the embodiment of the present application, the motion vector between the first frame image and the second frame image can be understood as the motion vector of the matching opaque object in the first frame image and the second frame image.

[0052] After the matching opaque objects in the first frame image and the second frame image are determined, the motion vectors of the matching opaque objects in the first frame image and the second frame image may be rendered through a complete graphics rendering pipeline, including vertex processing and fragment processing.

[0053] It is understandable that each polygon that makes up polygon modeling has multiple vertices (Vertex). These vertices have a coordinate in three-dimensional space. However, the screen of an electronic device is two-dimensional, so when determining the current viewing angle, the positions of these vertices in three-dimensional space need to be converted to the two-dimensional space of the screen. This conversion operation is called vertex processing.

[0054] Fragment processing refers to coloring pixels based on the color, transparency and other information of each pixel.

[0055] As a method, after obtaining the motion vector between the first frame image and the second frame image, the motion vector between the two frames can be written into a frame buffer (FrameBuffer) for storage. Further, the value of the FrameBuffer can be dumped into a 32-bit floating point type and converted into a PNG format of the motion vector in the x and y directions.

[0056] A motion vector determination method provided in the present application determines matching opaque objects in two frames of images based on the drawing call instructions corresponding to each frame of image, which can improve the accuracy of determining opaque objects. On the basis of improving the accuracy of determining opaque objects, the calculation accuracy of the motion vector between the two frames of image can be further improved.

[0057] See also Figure 4 , an embodiment of the present application provides a method for determining a motion vector, the method comprising:

[0058] Step S210: Acquire a first frame image and a second frame image that are consecutive in time.

[0059] Step S220: determining a first draw call instruction corresponding to the first frame image, and a second draw call instruction corresponding to the second frame image, wherein the first draw call instruction is a draw call instruction corresponding to an opaque object in the first frame image, and the second draw call instruction is a draw call instruction corresponding to an opaque object in the second frame image.

[0060] Step S230: obtaining a plurality of first matching features corresponding to the first drawing call instruction, wherein each of the first matching features is a specific feature obtained from a call stack corresponding to the first drawing call instruction through a preset sliding window, and a call stack of the preset sliding window corresponds to one first matching feature.

[0061] In the embodiment of the present application, the preset sliding window is a sliding window size preset for processing the call stack corresponding to the drawing call instruction, and the size of the preset sliding window can be determined according to the number of call functions required for rendering an opaque object. The call stack corresponding to the first drawing call instruction refers to the historical record of function calls during the execution of the program, which records the order of all function calls from the start of the program to the current execution point, as well as all the caches and parameters required in the OpenGL command. Among them, all the caches and parameters required in the OpenGL command can be understood as the motion vector calculation function required by the call stack, which is used to determine the vertex data of each object and the position of each pixel in the image, etc.

[0062] The call stack corresponding to the first drawing call instruction can be understood as a function sequence, that is, a function sequence composed of the call stacks corresponding to all the first drawing call instructions. The first matching feature may include vertex buffer index, vertex attribute parameters, etc.; it can be understood that since the size of the preset sliding window is determined according to the number of call functions required to render an opaque object, a first matching feature in the embodiment of the present application can be understood as a matching feature corresponding to an opaque object in the first frame image.

[0063] In the embodiment of the present application, a call stack of a preset sliding window length corresponds to a first matching feature. Optionally, the preset sliding window can slide according to a preset step length, so that the call stack corresponding to the first drawing call instruction can be divided into multiple call stacks of the preset sliding window length, and then a specific feature can be obtained from each call stack of the preset sliding window length to obtain multiple first matching features.

[0064] Step S240: obtaining a plurality of second matching features corresponding to the second drawing call instruction, wherein each second matching feature is a specific feature obtained from a call stack corresponding to the second drawing call instruction through the preset sliding window, and a call stack of the preset sliding window corresponds to one second matching feature.

[0065] In the embodiment of the present application, the preset sliding window is a preset sliding window size for processing a call stack corresponding to a drawing call instruction, and the size of the preset sliding window can be determined according to the number of call functions required for rendering an opaque object. Optionally, the size of the preset sliding window in the call stack for the first drawing call instruction and the size of the preset sliding window in the call stack for the second drawing call instruction can be set to the same size; or, the size of the preset sliding window in the call stack for the first drawing call instruction and the size of the preset sliding window in the call stack for the second drawing call instruction can both be determined according to the number of rendering functions required for an opaque object.

[0066] Similarly, the call stack corresponding to the second drawing call instruction refers to the history of function calls during program execution, which records the order of all function calls from the start of program execution to the current execution point, as well as all buffers and parameters required in the OpenGL command. Among them, all buffers and parameters required in the OpenGL command can be understood as motion vector calculation functions, which are used to determine the vertex data of each object and the position of each pixel in the image.

[0067] The call stack corresponding to the second drawing call instruction can also be understood as a function sequence, that is, a function sequence composed of the call stacks corresponding to all the second drawing call instructions. The second matching feature may include vertex buffer index, vertex attribute parameters, etc.; it can be understood that since the size of the preset sliding window is determined according to the number of call functions required to render an opaque object, a second matching feature in the embodiment of the present application can be understood as a matching feature corresponding to an opaque object in the second frame image.

[0068] In the embodiment of the present application, a call stack of a preset sliding window length corresponds to a second matching feature. Optionally, the preset sliding window can slide according to a preset step length, so that the call stack corresponding to the second drawing call instruction can be divided into multiple call stacks of the preset sliding window length, and then a specific feature can be obtained from each call stack of the preset sliding window length to obtain multiple second matching features.

[0069] Step S250: Based on the multiple first matching features and the multiple second matching features, determine the matching opaque objects in the first frame image and the second frame image.

[0070] In an embodiment of the present application, after obtaining multiple first matching features and multiple second matching features, the multiple first matching features and multiple second matching features can be matched in sequence, so that the matching opaque objects in the first frame image and the second frame image can be determined according to the matching results.

[0071] As one approach, determining the matching opaque objects in the first frame image and the second frame image based on the multiple first matching features and the multiple second matching features includes: respectively acquiring the similarities between the multiple first matching features and the multiple second matching features to obtain multiple similarities corresponding to each of the first matching features; if the maximum similarity among the multiple similarities corresponding to each of the first matching features is greater than a preset similarity, determining the object corresponding to the maximum similarity as the matching opaque object in the first frame image and the second frame image, wherein a drawing call instruction is used to instruct the drawing of an object.

[0072] The preset similarity is a preset similarity indicating that two opaque objects are the same object. For example, the preset similarity can be set to 95%.

[0073] Specifically, multiple first matching features are matched with multiple second matching features respectively to obtain the similarity between each first matching feature and each second matching feature, and then based on the similarity between each first matching feature and each second matching feature, the opaque object that matches the opaque object corresponding to each first matching feature can be determined from the opaque objects included in the second frame image.

[0074] Further, after obtaining multiple similarities corresponding to each first matching feature, the multiple similarities corresponding to each first matching feature can be compared to determine the maximum similarity among the multiple similarities corresponding to each first matching feature, that is, to obtain the maximum similarity corresponding to each first matching feature. After determining the maximum similarity corresponding to each first matching feature, the maximum similarity corresponding to each first matching feature can be compared with a preset similarity. If the maximum similarity corresponding to a first matching feature is greater than the preset similarity, it can be determined that the opaque object in the first frame image and the opaque object in the second frame image corresponding to the maximum similarity are matched opaque objects.

[0075] For example, Figure 5 As shown, the multiple first matching features are matching feature 1, matching feature 2, matching feature 3 and matching feature 4, and the multiple first matching features correspond to opaque object 1, opaque object 2, opaque object 3 and opaque object 4 in the first frame image respectively; the multiple second matching features are matching feature a, matching feature b, matching feature c and matching feature d, and the multiple second matching features correspond to opaque object a, opaque object b, opaque object c and opaque object d in the second frame image respectively.

[0076] Matching feature 1, matching feature 2, matching feature 3 and matching feature 4 with matching feature a, matching feature b, matching feature c and matching feature d are matched respectively to obtain similarity 1 between matching feature 1 and matching feature a, similarity 2 between matching feature 1 and matching feature b, similarity 3 between matching feature 1 and matching feature c, and similarity 4 between matching feature 1 and matching feature d; similarity 5 between matching feature 2 and matching feature a, similarity 6 between matching feature 2 and matching feature b, similarity 7 between matching feature 2 and matching feature c, and similarity 8 between matching feature 2 and matching feature d; similarity 9 between matching feature 3 and matching feature a, similarity 10 between matching feature 3 and matching feature b, similarity 11 between matching feature 3 and matching feature c, and similarity 12 between matching feature 3 and matching feature d; similarity 13 between matching feature 4 and matching feature a, similarity 14 between matching feature 4 and matching feature b, similarity 15 between matching feature 4 and matching feature c, and similarity 16 between matching feature 4 and matching feature d.

[0077] The maximum similarity among similarity 1, similarity 2, similarity 3 and similarity 4 corresponding to matching feature 1, the maximum similarity among similarity 5, similarity 6, similarity 7 and similarity 8 corresponding to matching feature 2, the maximum similarity among similarity 9, similarity 10, similarity 11 and similarity 12 corresponding to matching feature 3, and the maximum similarity among similarity 13, similarity 14, similarity 15 and similarity 16 corresponding to matching feature 4 are respectively compared with preset similarities to determine the opaque objects that match between the first frame image and the second frame image.

[0078] Exemplarily, the maximum similarity among similarity 1, similarity 2, similarity 3 and similarity 4 corresponding to matching feature 1 is determined. If the maximum similarity is similarity 3, and similarity 3 is greater than the preset similarity, it can be determined that the opaque object 1 and the opaque object c corresponding to similarity 3 are matching opaque objects.

[0079] Step S260: determining a motion vector between the first frame image and the second frame image based on matching opaque objects in the first frame image and the second frame image.

[0080] A motion vector determination method provided in the present application determines matching opaque objects in two frames of images based on the drawing call instructions corresponding to each frame of image, which can improve the accuracy of determining opaque objects. On the basis of improving the accuracy of determining opaque objects, the calculation accuracy of the motion vector between the two frames of image can be further improved.

[0081] See also Figure 6 , an embodiment of the present application provides a method for determining a motion vector, the method comprising:

[0082] Step S310: Obtain the capture file corresponding to the target application.

[0083] In an embodiment of the present application, the target application is a determined application that needs to calculate motion vectors. The target application can be a game application or a video playback application, which is not specifically limited here. Among them, the capture file corresponding to the target application is a capture file obtained by capturing continuous frame images into a single file through FrameProfiler when the target application is in a running state. Among them, FrameProfiler capture is based on the image context, and the frame is based on the exchange of the buffer surface. The frame here is based on the exchange of the buffer surface, which means that when each frame is rendered to the screen, the rendering data will be switched from the background buffer to the surface buffer.

[0084] Step S320: parsing the captured file through a frame parser to obtain a plurality of continuous frames of images and a drawing call instruction sequence corresponding to each of the plurality of frames of images.

[0085] In an embodiment of the present application, after obtaining a capture file of a target application, the capture file can be parsed by a frame parser in a preprocessing module to obtain a plurality of frame images, a drawing call instruction sequence corresponding to each of the plurality of frame images, and a motion vector calculation required for a call stack corresponding to each drawing call instruction.

[0086] Specifically, when a capture file is opened, the relevant modules of FrameProfiler will be executed to read the information in the capture file. After organizing the write file, the initial stream of the saved OpenGL function is first read, resources are created and initialized, and the initial state of the frame is set. Next, FrameProfiler processes other OpenGL events within the capture range, which are real functional image rendering draw calls.

[0087] The preprocessing module is triggered at the beginning of the capture range. Since FrameProfiler saves all frame information in the same capture file, a frame parser is needed. This frame parser is designed to automatically parse each frame in the capture file based on the specific signature of the capture application and avoid calling UI draw in the game. When processing draw calls, the draw calls and related call stacks are recorded, and the designed data structure is used to store the parameters.

[0088] Step S330: acquiring a first frame image and a second frame image that are consecutive in time from the multiple consecutive frame images.

[0089] In the embodiment of the present application, after obtaining multiple frames of images and the drawing call instruction sequences corresponding to the multiple frames of images, two consecutive frames of images, namely the first frame of image and the second frame of image, can be arbitrarily selected from them. At this time, when the first frame of image and the second frame of image are obtained, the drawing call instruction sequences and call stacks corresponding to the first frame of image and the second frame of image are also obtained.

[0090] Step S340: determining a first transparent object from objects included in the first frame image.

[0091] It will be appreciated that motion vectors are only applied to opaque objects to avoid visual artifacts, so transparent objects must not be taken into account when calculating motion vectors.

[0092] In an embodiment of the present application, after obtaining the first frame image and the second frame image, and the drawing call instruction sequence and the call stack corresponding to the first frame image and the second frame image respectively, the first transparent object included in the first frame image can be determined based on the drawing call sequence and the call stack corresponding to the first frame image.

[0093] Specifically, the object starting from glEnable(GL_BLEND) and ending with glDisable(GL_BLEND) in the drawing call sequence and the call stack corresponding to the first frame image may be determined as the first transparent object.

[0094] Step S350: deleting the drawing call instruction of the first transparent object in the drawing call instruction sequence corresponding to the first frame image from the cache, and obtaining the first drawing call instruction corresponding to the first frame image.

[0095] In the embodiment of the present application, after determining the first transparent object in the first frame image, the draw call corresponding to the first transparent object in the draw call instruction sequence corresponding to the first frame image can be deleted from the cache to obtain the first draw call instruction. At the same time, the call stack of the draw call corresponding to the first transparent object can also be deleted from the cache. That is, the objects starting from glEnable(GL_BLEND) to ending with glDisable(GL_BLEND) in the draw call instruction sequence corresponding to the first frame image can be deleted.

[0096] Step S360: determining a second transparent object from objects included in the second frame image.

[0097] In an embodiment of the present application, after obtaining the first frame image and the second frame image, and the drawing call instruction sequence and the call stack corresponding to the first frame image and the second frame image respectively, the second transparent object included in the second frame image can be determined based on the drawing call sequence and the call stack corresponding to the second frame image.

[0098] Specifically, the object starting from glEnable(GL_BLEND) and ending with glDisable(GL_BLEND) in the drawing call sequence and the call stack corresponding to the second frame image may be determined as the second transparent object.

[0099] Step S370: Delete the draw call instruction of the second transparent object in the draw call instruction sequence corresponding to the second frame image, and obtain the second draw call instruction corresponding to the second frame image.

[0100] In the embodiment of the present application, after the second transparent object in the second frame image is determined, the draw call corresponding to the second transparent object in the draw call instruction sequence corresponding to the second frame image can be deleted from the cache to obtain the second draw call instruction. At the same time, the call stack of the draw call corresponding to the second transparent object can also be deleted from the cache. That is, the objects starting from glEnable(GL_BLEND) to ending with glDisable(GL_BLEND) in the draw call instruction sequence corresponding to the second frame image can be deleted.

[0101] Step S380: determining matching opaque objects in the first frame image and the second frame image based on the first drawing call instruction and the second drawing call instruction.

[0102] In the embodiment of the present application, after deleting the drawing calls corresponding to the transparent objects, a sliding window matching algorithm is proposed based on the cleaned drawing call instructions. Since the frames are captured at a reasonable time interval, usually above 30fps, the scene will not change significantly, and the drawing call sequence follows the compilation order of the game execution, the matching opaque objects in the first frame image and the second frame image can be determined by using the sliding window to search for local maximum matching.

[0103] Step S390: determining a motion vector between the first frame image and the second frame image based on matching opaque objects in the first frame image and the second frame image.

[0104] A motion vector determination method provided in the present application determines matching opaque objects in two frames of images based on the drawing call instructions corresponding to each frame of image, which can improve the accuracy of determining opaque objects. On the basis of improving the accuracy of determining opaque objects, the calculation accuracy of the motion vector between the two frames of image can be further improved.

[0105] See also Figure 7 , an embodiment of the present application provides a method for determining a motion vector, the method comprising:

[0106] Step S410: Acquire a first frame image and a second frame image that are consecutive in time.

[0107] Step S420: determining a first draw call instruction corresponding to the first frame image, and a second draw call instruction corresponding to the second frame image, wherein the first draw call instruction is a draw call instruction corresponding to an opaque object in the first frame image, and the second draw call instruction is a draw call instruction corresponding to an opaque object in the second frame image.

[0108] Step S430: determining matching opaque objects in the first frame image and the second frame image based on the first drawing call instruction and the second drawing call instruction.

[0109] Step S440: performing vertex processing on the opaque objects that match the first frame image and the second frame image through a vertex shader to determine the world positions of the opaque objects that match the first frame image and the second frame image.

[0110] In the embodiment of the present application, vertex processing uses the vertices and MVP (ModelView Matrix) matrix of the first frame image and the second frame image as the input of the vertex shader, and takes the world position (gl_Position in OpenGL) of the two frame objects as the output. The two frame objects refer to the matching opaque objects in the first frame image and the second frame image; the vertices can be determined by the motion vector calculation function required by the call stack corresponding to each opaque object.

[0111] Step S450: Based on the world positions of the matching opaque objects in the first frame image and the second frame image, fragment processing is performed on the matching opaque objects in the first frame image and the second frame image to obtain a motion vector between the first frame image and the second frame image.

[0112] In the embodiment of the present application, the fragment processing uses the output of the fixed point shader as the input of the fragment shader, calculates the normalized motion vector by defining an equation, and sets it as the fragment color output. Taking the OpenGL fragment shader as an example, FragColor = vec4 (u_pos-u_prev_pos, v_pos-v_prev_pos, 0.0, 1.0).

[0113] After obtaining the motion vector between the first frame image and the second frame image, the motion vector between the two frames is written into the FrameBuffer, and then the value of the FrameBuffer can be dumped as a 32-bit floating point type and converted into a PNG format of the motion vector in the x and y directions.

[0114] As a method, after step S450, the following steps are further included: verifying the motion vector between the first frame image and the second frame image; if the verification is successful, storing the motion vector between the first frame image and the second frame image.

[0115] After the motion vector between the first frame image and the second frame image is obtained, the calculated motion vector may be verified to verify its correctness.

[0116] Specifically, the calculated motion vector may be compared with the motion vector obtained through the Unity API. If the two are the same, it is proved that the calculated motion vector is correct.

[0117] Optionally, the process described in step S410 to step S450 in the embodiment of the present application can be performed offline, that is, the entire calculation process is performed at the back end.

[0118] A motion vector determination method provided in the present application determines matching opaque objects in two frames of images based on the drawing call instructions corresponding to each frame of image, which can improve the accuracy of determining opaque objects. On the basis of improving the accuracy of determining opaque objects, the calculation accuracy of the motion vector between the two frames of image can be further improved.

[0119] See also Figure 8 , an embodiment of the present application provides a motion vector determination device 500, the device 500 comprising:

[0120] The image acquisition unit 510 is used to acquire a first frame image and a second frame image that are continuous in time.

[0121] As a method, the image acquisition unit 510 is specifically used to obtain a capture file corresponding to a target application; parse the capture file through a frame parser to obtain a continuous multi-frame image and a drawing call instruction sequence corresponding to each of the multi-frame images; and obtain a first frame image and a second frame image that are continuous in time from the continuous multi-frame images.

[0122] The instruction determination unit 520 is used to determine a first drawing call instruction corresponding to the first frame image, and a second drawing call instruction corresponding to the second frame image, wherein the first drawing call instruction is a drawing call instruction corresponding to an opaque object in the first frame image, and the second drawing call instruction is a drawing call instruction corresponding to an opaque object in the second frame image.

[0123] As a method, the instruction determination unit 520 is specifically used to determine a first transparent object from objects included in the first frame image; delete the drawing call instruction of the first transparent object in the drawing call instruction sequence corresponding to the first frame image from the cache, and obtain the first drawing call instruction corresponding to the first frame image; determine a second transparent object from objects included in the second frame image; delete the drawing call instruction of the second transparent object in the drawing call instruction sequence corresponding to the second frame image, and obtain the second drawing call instruction corresponding to the second frame image.

[0124] The object determining unit 530 is configured to determine matching opaque objects in the first frame image and the second frame image based on the first drawing call instruction and the second drawing call instruction.

[0125] As a method, the object determination unit 530 is specifically used to obtain multiple first matching features corresponding to the first drawing call instruction, each of the first matching features is a specific feature obtained from a call stack corresponding to the first drawing call instruction through a preset sliding window, and a call stack of the preset sliding window corresponds to one first matching feature; obtain multiple second matching features corresponding to the second drawing call instruction, each of the second matching features is a specific feature obtained from a call stack corresponding to the second drawing call instruction through the preset sliding window, and a call stack of the preset sliding window corresponds to one second matching feature; based on the multiple first matching features and the multiple second matching features, determine the matching opaque objects in the first frame image and the second frame image.

[0126] Furthermore, the object determination unit 530 is specifically used to respectively obtain the similarities between the multiple first matching features and the multiple second matching features, and obtain multiple similarities corresponding to each of the first matching features; if the maximum similarity among the multiple similarities corresponding to each of the first matching features is greater than a preset similarity, the object corresponding to the maximum similarity is determined as an opaque object that matches the first frame image and the second frame image, wherein a drawing call instruction is used to instruct the drawing of an object.

[0127] The motion vector determining unit 540 is configured to determine a motion vector between the first frame image and the second frame image based on matching opaque objects in the first frame image and the second frame image.

[0128] As a method, the motion vector determination unit 540 is specifically used to perform vertex processing on the opaque objects that match the first frame image and the second frame image through a vertex shader to determine the world positions of the opaque objects that match the first frame image and the second frame image; based on the world positions of the opaque objects that match the first frame image and the second frame image, fragment processing is performed on the opaque objects that match the first frame image and the second frame image to obtain the motion vector between the first frame image and the second frame image.

[0129] Optional, such as Fig. 9 As shown, the device 500 also includes:

[0130] The verification unit 550 is used to verify the motion vector between the first frame image and the second frame image; if the verification is successful, the motion vector between the first frame image and the second frame image is stored.

[0131] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0132] The following will be combined Fig.10 An electronic device provided by the present application is described.

[0133] See also Fig.10 Based on the above-mentioned motion vector determination method and device, the embodiment of the present application also provides another electronic device 800 that can execute the above-mentioned motion vector determination method. The electronic device 800 includes one or more (only one is shown in the figure) processors 802, a memory 804 and a network module 806 that are coupled to each other. Among them, the memory 804 stores a program that can execute the content of the above-mentioned embodiment, and the processor 802 can execute the program stored in the memory 804.

[0134] Among them, the processor 802 may include one or more processing cores. The processor 802 uses various interfaces and lines to connect various parts of the entire electronic device 800, and executes various functions and processes data of the electronic device 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 804, and calling data stored in the memory 804. Optionally, the processor 802 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 802 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 802, but may be implemented separately through a communication chip.

[0135] The memory 804 may include a random access memory (RAM) or a read-only memory (ROM). The memory 804 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 804 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 800 during use.

[0136] The network module 806 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices, such as communicating with electronic devices. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. The network module 806 may communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices via wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 806 may exchange information with a base station.

[0137] Please refer to Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0138] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0139] The present application provides a method, device, electronic device and storage medium for determining a motion vector. First, a first frame image and a second frame image that are continuous in time are obtained, and a first drawing call instruction corresponding to the first frame image and a second drawing call instruction corresponding to the second frame image are determined, wherein the first drawing call instruction is a drawing call instruction corresponding to an opaque object in the first frame image, and the second drawing call instruction is a drawing call instruction corresponding to an opaque object in the second frame image. Then, based on the first drawing call instruction and the second drawing call instruction, matching opaque objects in the first frame image and the second frame image are determined. Finally, based on the matching opaque objects in the first frame image and the second frame image, a motion vector between the first frame image and the second frame image is determined. Through the above method, matching opaque objects in two frames of images are determined based on the drawing call instruction corresponding to each frame image, so that the determination accuracy of opaque objects can be improved. On the basis of improving the determination accuracy of opaque objects, the calculation accuracy of motion vectors between two frames of images can be further improved.

[0140] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for determining a motion vector, characterized in that: The method comprises: Acquire a first frame image and a second frame image that are continuous in time; Determine a first draw call instruction corresponding to the first frame image, and a second draw call instruction corresponding to the second frame image, wherein the first draw call instruction is a draw call instruction corresponding to an opaque object in the first frame image, and the second draw call instruction is a draw call instruction corresponding to an opaque object in the second frame image; Determine, based on the first drawing call instruction and the second drawing call instruction, matching opaque objects in the first frame image and the second frame image; Based on the matching opaque objects in the first frame image and the second frame image, a motion vector between the first frame image and the second frame image is determined.

2. The method according to claim 1, characterized in that The determining, based on the first drawing call instruction and the second drawing call instruction, matching opaque objects in the first frame image and the second frame image includes: Acquire a plurality of first matching features corresponding to the first drawing call instruction, where each of the first matching features is a specific feature acquired from a call stack corresponding to the first drawing call instruction through a preset sliding window, and a call stack of the preset sliding window corresponds to one first matching feature; Acquire a plurality of second matching features corresponding to the second drawing call instruction, where each of the second matching features is a specific feature acquired from a call stack corresponding to the second drawing call instruction through the preset sliding window, and a call stack of the preset sliding window corresponds to one second matching feature; Based on the multiple first matching features and the multiple second matching features, matching opaque objects in the first frame image and the second frame image are determined.

3. The method according to claim 2, characterized in that The determining, based on the plurality of first matching features and the plurality of second matching features, the matching opaque objects in the first frame image and the second frame image comprises: Respectively obtaining similarities between the plurality of first matching features and the plurality of second matching features, and obtaining a plurality of similarities corresponding to each of the first matching features; If the maximum similarity among the multiple similarities corresponding to each of the first matching features is greater than a preset similarity, the object corresponding to the maximum similarity is determined as an opaque object that matches in the first frame image and the second frame image, wherein one drawing call instruction is used to instruct drawing one object.

4. The method according to claim 1, characterized in that: The acquiring of a first frame image and a second frame image that are consecutive in time comprises: Get the capture file corresponding to the target application; Parsing the captured file by a frame parser to obtain a plurality of continuous frames of images and a drawing call instruction sequence corresponding to each of the plurality of frames of images; A first frame image and a second frame image which are continuous in time are acquired from the continuous multiple frame images.

5. The method according to claim 4, characterized in that The determining a first drawing call instruction corresponding to the first frame image and a second drawing call instruction corresponding to the second frame image includes: Determining a first transparent object from objects included in the first frame of image; In the drawing call instruction sequence corresponding to the first frame image, the drawing call instruction of the first transparent object is deleted from the cache to obtain a first drawing call instruction corresponding to the first frame image; determining a second transparent object from objects included in the second frame of image; The draw call instruction of the second transparent object is deleted from the draw call instruction sequence corresponding to the second frame image to obtain a second draw call instruction corresponding to the second frame image.

6. The method according to claim 1, characterized in that The determining of the motion vector between the first frame image and the second frame image based on the matching opaque objects in the first frame image and the second frame image includes: Performing vertex processing on the opaque objects that match the first frame image and the second frame image by a vertex shader to determine the world positions of the opaque objects that match the first frame image and the second frame image; Based on the world positions of the matching opaque objects in the first frame image and the second frame image, fragment processing is performed on the matching opaque objects in the first frame image and the second frame image to obtain a motion vector between the first frame image and the second frame image.

7. The method according to claim 6, characterized in that After determining the motion vector between the first frame image and the second frame image based on the matching opaque objects in the first frame image and the second frame image, the method further includes: Verifying a motion vector between the first frame image and the second frame image; If the verification is successful, the motion vector between the first frame image and the second frame image is stored.

8. A motion vector determination device, characterized in that: The device comprises: An image acquisition unit, used for acquiring a first frame image and a second frame image which are continuous in time; an instruction determining unit, configured to determine a first drawing call instruction corresponding to the first frame image, and a second drawing call instruction corresponding to the second frame image, wherein the first drawing call instruction is a drawing call instruction corresponding to an opaque object in the first frame image, and the second drawing call instruction is a drawing call instruction corresponding to an opaque object in the second frame image; an object determining unit, configured to determine matching opaque objects in the first frame image and the second frame image based on the first drawing call instruction and the second drawing call instruction; The motion vector determining unit is used to determine the motion vector between the first frame image and the second frame image based on the matching opaque objects in the first frame image and the second frame image.

9. An electronic device, characterized in that: The method comprises one or more processors; one or more programs are stored in the memory and configured to execute the method according to any one of claims 1 to 7 by the one or more processors.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images