Motion estimation based on region discontinuities

Abstract

The present disclosure relates to a method and device for image or frame processing, including an apparatus, e.g., a GPU. In some aspects, the apparatus may identify a discontinuity between at least one first region of a first frame and at least one second region of a second frame, where the at least one first region corresponds to the at least one second region. The apparatus may also determine whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold. The apparatus may also skip motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. patent application Ser. No. 17 / 087,528, filed November 2, 2020, entitled "METHODS AND APPARATUS FOR MOTION ESTIMATION BASED ON REGION DISCONTINUITY," the entire contents of which are expressly incorporated herein by reference.

[0002] The present disclosure relates generally to processing systems, and more particularly to one or more techniques for image or frame processing. [Background technology]

[0003] Introduction

[0003] Computing devices often utilize graphics processing units (GPUs) to accelerate the rendering of graphical data for display. Such computing devices may include, for example, computer workstations, mobile phones such as so-called smartphones, embedded systems, personal computers, tablet computers, and video game consoles. A GPU executes a graphics processing pipeline that includes one or more processing stages that work together to execute graphics processing commands and output frames. A central processing unit (CPU) may control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern CPUs are generally capable of concurrently executing multiple applications, each of which may require the use of the GPU during execution. Devices that provide content for visual presentation on a display generally include a GPU.

[0004]

[0004] Typically, a device's GPU is configured to perform processes in a graphics processing pipeline. However, with the advent of wireless communications and smaller handheld devices, an increased need for improved graphics processing has arisen. Summary of the Invention

[0005] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, nor is it intended to identify key elements of all aspects or delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0006]

[0006] In one aspect of the present disclosure, a method, a computer-readable medium, and an apparatus are provided. The apparatus may be a GPU, a CPU, an image processor, a frame processor, a display processing unit (DPU), and / or any apparatus capable of performing image or frame processing. The apparatus may receive a second frame after a first frame is received. The apparatus may also compare at least one first region of the first frame with at least one second region of the second frame. The apparatus may also identify a discontinuity between the at least one first region of the first frame and the at least one second region of the second frame, wherein the at least one first region corresponds to at least one second region, the first frame includes a plurality of first regions, the second frame includes a plurality of second regions, and the first frame and the second frame relate to a plurality of frames. Furthermore, the apparatus may determine whether a discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold. The apparatus may also skip motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold. Moreover, the apparatus may perform motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is less than the motion estimation threshold. The apparatus may also construct a third frame based on the performed motion estimation, where the constructed third frame may be associated with a frame extrapolation process.

[0007] The details of one or more examples of this disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present disclosure will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0008] [Figure 1]

[0008] A block diagram illustrating an example content generation system in accordance with one or more techniques of this disclosure. [Figure 2]

[0009] 1 illustrates an example GPU, in accordance with one or more techniques of this disclosure. [Figure 3]

[0010] 1A-1C illustrate exemplary images or surfaces in accordance with one or more techniques of the present disclosure. [Figure 4A]

[0011] 1 illustrates an example frame in accordance with one or more techniques of this disclosure. [Figure 4B] 1 illustrates an example frame in accordance with one or more techniques of this disclosure. [Figure 5]

[0012] 1 is an exemplary diagram of image processing in accordance with one or more techniques of this disclosure. [Figure 6]

[0013] 1 is an exemplary diagram of image processing in accordance with one or more techniques of this disclosure. [Figure 7]

[0014] 1 is an exemplary diagram of image processing components in accordance with one or more techniques of this disclosure. [Figure 8]

[0015] 1 is an example flowchart of an example method in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009]

[0016] Extended reality (XR), augmented reality (AR), or virtual reality (VR) applications may be utilized with some devices, e.g., mobile devices or smartphones, that may be under strict constraints on power and performance efficiency. To alleviate these constraints, motion estimation may be performed on previously rendered content and / or used to extrapolate frames. While motion estimation may work well at times, content may be discontinuous when transitioning between frames, such as from frame to frame. In these cases, performing motion estimation may result in inaccurate data based on the discontinuous content between frames, e.g., inaccurate or false motion estimation data. This inaccurate data may result in poor or incomplete frame extrapolation based on motion estimation. In some cases, motion estimation and / or frame extrapolation applied across these discontinuous frames may produce random or incorrect motion vectors. Furthermore, these random or incorrect motion vectors produced from discontinuous frames may cause strong, unintended distortions in the extrapolated frame or in some portions of the extrapolated frame. For example, if there are discontinuities between frames based on frame content or brightness, inaccurate motion estimation may be produced. Aspects of the present disclosure may skip or bypass performing motion estimation for some frames or regions of frames to avoid incorrect motion vectors and / or unintended distortions. For example, aspects of the present disclosure may identify discontinuities or differences between some frames or regions of frames. Furthermore, aspects of the present disclosure may determine whether the discontinuities or differences between some frames or regions of frames exceed a threshold, and then skip or bypass performing motion estimation if the discontinuities or differences exceed the threshold.

[0010]

[0017] Various aspects of the systems, devices, computer program products, and methods are described more fully below with reference to the accompanying drawings. However, the present disclosure may be embodied in many different forms and should not be construed as limited to any specific structure or function presented throughout this disclosure. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Based on the teachings herein, those skilled in the art will appreciate that the scope of the present disclosure covers any aspect of the systems, devices, computer program products, and methods disclosed herein, whether implemented independently of or in combination with other aspects of the present disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of aspects described herein. Furthermore, the scope of the present disclosure is intended to cover such apparatuses or methods practiced using other structure, functions, or structure and functions in addition to or other than the various aspects of the present disclosure described herein. Any aspect disclosed herein may be embodied by one or more elements of a claim.

[0011]

[0018] While various aspects are described herein, many variations and permutations of these aspects fall within the scope of the present disclosure. Some potential benefits and advantages of aspects of the present disclosure are described, but the scope of the present disclosure is not limited to any particular benefit, use, or purpose. Rather, aspects of the present disclosure are intended to be broadly applicable to different wireless technologies, system configurations, networks, and transmission protocols, some of which are illustrated by way of example in the figures and in the following description. The detailed description and drawings are merely illustrative rather than limiting of the present disclosure, the scope of which is defined by the appended claims and their equivalents.

[0012]

[0019] Several aspects are presented with respect to various apparatus and methods. These apparatus and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements may be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends on the particular application and design constraints imposed on the overall system.

[0013]

[0020] As an example, an element, or any portion of an element, or any combination of elements, may be implemented as a "processing system" including one or more processors (sometimes referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems-on-chips (SOCs), baseband processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gate logic, discrete hardware circuits, and other suitable hardware configured to perform various functions described throughout this disclosure. One or more processors in a processing system may execute software. Software may be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. The term application may refer to software. As described herein, one or more techniques may refer to an application, i.e., software configured to perform one or more functions. In such examples, an application may be stored in memory, e.g., a processor's on-chip memory, system memory, or any other memory. Hardware described herein, such as a processor, may be configured to execute the application. For example, an application may be described as including code that, when executed by the hardware, causes the hardware to perform one or more techniques described herein.As an example, hardware may access code from memory and execute the code accessed from memory to perform one or more techniques described herein. In some examples, components are identified in this disclosure. In such examples, the components may be hardware, software, or a combination thereof. The components may be separate components or subcomponents of a single component.

[0014]

[0021] Thus, in one or more examples described herein, the functions described may be implemented in hardware, software, or any combination thereof. If implemented in software, the functions may be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media. A storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may comprise random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the above types of computer-readable media, or any other medium that can be used to store computer-executable code in the form of instructions or data structures that can be accessed by a computer.

[0015]

[0022] Generally, this disclosure describes techniques for improving the rendering of graphical content and / or reducing the load on a processing unit, i.e., any processing unit configured to perform one or more techniques described herein, such as a GPU, having a graphics processing pipeline in a single device or multiple devices. For example, this disclosure describes techniques for graphics processing in any device that utilizes graphics processing. Other example benefits are described throughout this disclosure.

[0016]

[0023] Instances of the term “content” as used herein may refer to “graphical content,” “image,” and vice versa. This is true whether the term is used as an adjective, a noun, or other part of speech. In some examples, the term “graphical content” as used herein may refer to content produced by one or more processes of a graphics processing pipeline. In some examples, the term “graphical content” as used herein may refer to content produced by a processing unit configured to perform graphics processing. In some examples, the term “graphical content” as used herein may refer to content produced by a graphics processing unit.

[0017]

[0024] In some examples, the term "display content" as used herein may refer to content generated by a processing unit configured to perform display processing. In some examples, the term "display content" as used herein may refer to content generated by a display processing unit. Graphical content may be processed to become display content. For example, a graphics processing unit may output graphical content, such as a frame, to a buffer (sometimes referred to as a frame buffer). The display processing unit may read the graphical content, such as one or more frames from the buffer, and perform one or more display processing techniques thereon to generate the display content. For example, the display processing unit may be configured to perform compositing on one or more rendered layers to generate a frame. As another example, the display processing unit may be configured to composite, blend, or otherwise combine two or more layers together into a single frame. The display processing unit may be configured to perform scaling, e.g., upscaling or downscaling, on a frame. In some examples, a frame may refer to a layer. In other examples, a frame may refer to two or more layers that have already been blended together to form a frame; i.e., a frame includes two or more layers, and a frame including two or more layers may then be blended.

[0018]

[0025] FIG. 1 is a block diagram illustrating an example content generation system 100 configured to implement one or more techniques of the present disclosure. The content generation system 100 includes a device 104. The device 104 may include one or more components or circuits for performing various functions described herein. In some examples, one or more components of the device 104 may be components of a system-on-chip (SOC). The device 104 may include one or more components configured to perform one or more techniques of the present disclosure. In the illustrated example, the device 104 may include a processing unit 120, a content encoder / decoder 122, and a system memory 124. In some aspects, the device 104 may include several optional components, such as a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131. References to the display 131 may refer to one or more displays 131. For example, the display 131 may include a single display or multiple displays. The display 131 may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first display and the second display may receive different frames for presentation thereon. In other examples, the first display and the second display may receive the same frames for presentation thereon. In further examples, the results of the graphics processing may not be displayed on the device, e.g., the first and second displays may not receive frames for presentation thereon. Instead, the frames or the graphics processing results may be forwarded to another device. In some aspects, this may be referred to as split-rendering.

[0019]

[0026] Processing unit 120 may include internal memory 121. Processing unit 120 may be configured to perform graphics processing, such as in graphics processing pipeline 107. Content encoder / decoder 122 may include internal memory 123. In some examples, device 104 may include a display processor, such as display processor 127, to perform one or more display processing techniques on one or more frames generated by processing unit 120 before presentation by one or more displays 131. Display processor 127 may be configured to perform display processing. For example, display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by processing unit 120. One or more displays 131 may be configured to display or otherwise present frames processed by display processor 127. In some examples, one or more displays 131 may include one or more of a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0020]

[0027] Memory external to the processing unit 120 and the content encoder / decoder 122, such as system memory 124, may be accessible to the processing unit 120 and the content encoder / decoder 122. For example, the processing unit 120 and the content encoder / decoder 122 may be configured to read from and / or write to external memory, such as the system memory 124. The processing unit 120 and the content encoder / decoder 122 may be communicatively coupled to the system memory 124 over a bus. In some examples, the processing unit 120 and the content encoder / decoder 122 may be communicatively coupled to each other over a bus or a different connection.

[0021]

[0028] The content encoder / decoder 122 may be configured to receive graphical content from any source, such as the system memory 124 and / or the communication interface 126. The system memory 124 may be configured to store the received encoded or decoded graphical content. The content encoder / decoder 122 may be configured to receive the encoded or decoded graphical content in the form of coded pixel data, for example, from the system memory 124 and / or the communication interface 126. The content encoder / decoder 122 may be configured to encode or decode any graphical content.

[0022]

[0029] The internal memory 121 or the system memory 124 may include one or more volatile or non-volatile memory or storage devices. In some examples, the internal memory 121 or the system memory 124 may include RAM, SRAM, DRAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic or optical data media, or any other type of memory.

[0023]

[0030] Internal memory 121 or system memory 124 may be a non-transitory storage medium, according to some examples. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that internal memory 121 or system memory 124 is non-movable or that its contents are static. As one example, system memory 124 may be removed from device 104 and moved to another device. As another example, system memory 124 may not be removable from device 104.

[0024]

[0031] Processing unit 120 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, processing unit 120 may be integrated into the motherboard of device 104. In some examples, processing unit 120 may reside on a graphics card installed in a port in the motherboard of device 104, or in some cases may be embedded within a peripheral device configured to interoperate with device 104. Processing unit 120 may include one or more processors, such as one or more microprocessors, GPUs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the techniques are implemented partially in software, processing unit 120 may store instructions for the software in a suitable non-transitory computer-readable storage medium, such as internal memory 121, and execute those instructions in hardware using one or more processors to implement the techniques of the present disclosure. Any of the above, including hardware, software, a combination of hardware and software, etc., may be considered to be one or more processors.

[0025]

[0032] The content encoder / decoder 122 may be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 may be integrated into the motherboard of the device 104. The content encoder / decoder 122 may include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuitry, or any combination thereof. If the techniques are implemented partially in software, the content encoder / decoder 122 may store instructions for the software in a suitable non-transitory computer-readable storage medium, for example, internal memory 123, and may execute those instructions in hardware using one or more processors to implement the techniques of this disclosure. Any of the above, including hardware, software, a combination of hardware and software, etc., may be considered to be one or more processors.

[0026]

[0033] In some aspects, the content generation system 100 may include an optional communication interface 126. The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any receiving functionality described herein with respect to the device 104. Additionally, the receiver 128 may be configured to receive information, such as eye or head position information, rendering commands, or location information, from another device. The transmitter 130 may be configured to perform any transmitting functionality described herein with respect to the device 104. For example, the transmitter 130 may be configured to transmit information, which may include a request for content, to another device. The receiver 128 and the transmitter 130 may be combined into a transceiver 132. In such an example, the transceiver 132 may be configured to perform any receiving and / or transmitting functionality described herein with respect to the device 104.

[0027]

[0034] Referring again to FIG. 1 , in some aspects, graphics processing pipeline 107 may include a determination component 198 configured to receive a second frame after a first frame is received. The determination component 198 may also be configured to compare at least one first region of the first frame with at least one second region of the second frame. The determination component 198 may also be configured to identify a discontinuity between the at least one first region of the first frame and the at least one second region of the second frame, where the at least one first region corresponds to the at least one second region, the first frame includes multiple first regions, the second frame includes multiple second regions, and the first frame and the second frame relate to multiple frames. The determination component 198 may also be configured to determine whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold. The determination component 198 may also be configured to skip motion estimation for the at least one first region and the at least one second region when a discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold. The determination component 198 may also be configured to perform motion estimation for the at least one first region and the at least one second region when a discontinuity between the at least one first region and the at least one second region is less than a motion estimation threshold. The determination component 198 may also be configured to construct a third frame based on the performed motion estimation, where the constructed third frame may be associated with a frame extrapolation process.

[0028]

[0035] As described herein, a device, such as device 104, may refer to any device, apparatus, or system configured to implement one or more techniques described herein. For example, a device may be a server, a base station, a user equipment, a client device, a station, an access point, a computer, e.g., a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation, or a mainframe computer, an end product, an apparatus, a phone, a smartphone, a server, a video game platform or console, a handheld device, e.g., a portable video game device or personal digital assistant (PDA), a wearable computing device, e.g., a smartwatch, an augmented reality device, or a virtual reality device, a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, an in-car computer, any mobile device, any device configured to generate graphical content, or any device configured to implement one or more techniques described herein. While processes herein may be described as being performed by particular components (e.g., a GPU), in further embodiments, they may be implemented using other components (e.g., a CPU) in accordance with disclosed embodiments.

[0029]

[0036] A GPU can process multiple types of data or data packets in a GPU pipeline. For example, in some aspects, a GPU can process two types of data or data packets, e.g., context register packets and draw call data. A context register packet can be a set of global state information, e.g., information about global registers, shading programs, or constant data, that can adjust how a graphics context is processed. For example, a context register packet can include information about a color format. In some aspects of a context register packet, there can be a bit that indicates which workload belongs to a context register. Also, there can be multiple functions or programming running simultaneously and / or in parallel. For example, a function or programming can describe a certain operation, e.g., a color mode or color format. Thus, a context register can define multiple states of the GPU.

[0030]

[0037] The context state may be utilized to determine how individual processing units, e.g., a vertex fetcher (VFD), a vertex shader (VS), a shader processor, or a geometry processor, function and / or in which mode the processing units function. To do so, the GPU may use context registers and programming data. In some aspects, the GPU may generate workloads, e.g., vertex or pixel workloads, in a pipeline based on the context register definition of a mode or state. Some processing units, e.g., a VFD, may use these states to determine some functions, e.g., how vertices are assembled. Because these modes or states can change, the GPU may need to modify the corresponding context. Furthermore, the workload corresponding to a mode or state may follow the changing mode or state.

[0031]

[0038] 2 illustrates an example GPU 200 in accordance with one or more techniques of this disclosure. As shown in FIG. 2, GPU 200 includes a command processor (CP) 210, a draw call packet 212, a VFD 220, a VS 222, a vertex cache (VPC) 224, a triangle setup engine (TSE) 226, a rasterizer (RAS) 228, a Z process engine (ZPE) 230, a pixel interpolator (PI) 232, a fragment shader (FS) 234, a render backend (RB) 236, an L2 cache (UCHE) 238, and a system memory 240. While FIG. 2 illustrates GPU 200 including processing units 220-238, GPU 200 may include several additional processing units. Additionally, processing units 220-238 are merely examples, and any combination or order of processing units may be used by a GPU in accordance with this disclosure. GPU 200 also includes a command buffer 250, a context register packet 260, and a context state 261.

[0032]

[0039] 2, the GPU may utilize a CP, e.g., CP 210, or a hardware accelerator, to parse the command buffer into context register packets, e.g., context register packet 260, and / or draw call data packets, e.g., draw call packet 212. CP 210 may then send context register packet 260 or draw call data packet 212 through separate paths to processing units or blocks in the GPU. Furthermore, command buffer 250 may alternate between different states of context registers and draw calls. For example, the command buffer may be structured in the following manner: context register for context N, draw call(s) for context N, context register for context N+1, and draw call(s) for context N+1.

[0033]

[0040] A GPU can render an image in a variety of different ways. In some instances, a GPU can render an image using tiled rendering or tiled rendering. In a tiled rendering GPU, an image can be divided or separated into different sections or tiles. After dividing the image, each section or tile can be rendered separately. A tiled rendering GPU can divide a computer graphics image into a grid format such that each portion of the grid, i.e., a tile, is rendered separately. In some aspects, during a binning pass, an image can be divided into different bins or tiles. In some aspects, during the binning pass, a visibility stream can be constructed in which visible primitives or draw calls can be identified.

[0034]

[0041] In some aspects, a GPU can apply drawing or rendering processes to different bins or tiles. For example, a GPU can render to a bin and perform all drawing for the primitives or pixels in the bin. During the process of rendering to a bin, a render target may be in GMEM. In some cases, after rendering to a bin, the contents of the render target may be moved to system memory, and the GMEM may be freed to render the next bin. Additionally, a GPU can render to another bin and perform drawing for the primitives or pixels in that bin. Thus, in some aspects, there may be a small number of bins, e.g., four bins, that cover all of the drawing in a surface. Furthermore, a GPU can cycle through all of the drawing in a bin but perform drawing for the draw calls that are visible, i.e., the draw calls that include visible geometry. In some aspects, a visibility stream may be generated, e.g., in a binning pass, to determine visibility information for each primitive in an image or scene. For example, this visibility stream may identify whether a primitive is visible or not. In some aspects, this information may be used, for example, to remove non-visible primitives in a rendering pass, and at least some of the primitives identified as visible may be rendered in a rendering pass.

[0035]

[0042] In some aspects of tile rendering, there may be multiple processing phases or passes. For example, rendering may be performed in two passes, e.g., a visibility or bin visibility pass and a rendering or bin rendering pass. During the visibility pass, the GPU may input the rendering workload, record the positions of primitives or triangles, and then determine which primitives or triangles fall into which bins or areas. In some aspects of the visibility pass, the GPU may also identify or mark the visibility of each primitive or triangle in the visibility stream. During the rendering pass, the GPU may input the visibility stream and process one bin or area at a time. In some aspects, the visibility stream may be analyzed to determine which primitives, or vertices of primitives, are visible or invisible. Thus, the primitives, or vertices of primitives, that are visible may be processed. By doing so, the GPU may reduce the unnecessary workload of processing or rendering invisible primitives or triangles.

[0036]

[0043] In some aspects, during the visibility pass, some types of primitive geometry, e.g., position-only geometry, may be processed. Furthermore, depending on the position or location of the primitive or triangle, the primitive may be classified into different bins or areas. In some cases, classifying primitives or triangles into different bins may be performed by determining visibility information for these primitives or triangles. For example, the GPU may determine or write visibility information for each primitive in each bin or area, e.g., in system memory. This visibility information may be used to determine or generate a visibility stream. In the rendering pass, the primitives in each bin may be rendered separately. In these cases, the visibility stream may be fetched from memory to be used to drop primitives that are not visible for that bin.

[0037]

[0044] Some aspects of a GPU or GPU architecture can offer several different options for rendering, for example, software rendering and hardware rendering. In software rendering, the driver or CPU can replicate the entire frame geometry by processing each view once. Furthermore, several different states can be changed depending on the view. Thus, in software rendering, the software can replicate the entire workload by modifying several states that can be utilized to render for each viewpoint in the image. In some aspects, there can be an increased amount of overhead because the GPU may be submitting the same workload multiple times for each viewpoint in the image. In hardware rendering, the hardware or GPU can be responsible for replicating or processing the geometry for each viewpoint in the image. Thus, the hardware can manage the replication or processing of primitives or triangles for each viewpoint in the image.

[0038]

[0045] FIG. 3 illustrates an image or surface 300 that includes a plurality of primitives divided into a plurality of bins. As shown in FIG. 3, image or surface 300 includes area 302 that includes primitives 321, 322, 323, and 324. Primitives 321, 322, 323, and 324 are divided or placed into different bins, e.g., bins 310, 311, 312, 313, 314, and 315. FIG. 3 illustrates an example of tiled rendering using multiple viewpoints for primitives 321-324. For example, primitives 321-324 are in first viewpoint 350 and second viewpoint 351. Thus, a GPU processing or rendering image or surface 300 that includes area 302 can utilize multiple viewpoints or multi-view rendering.

[0039]

[0046] As shown herein, a GPU or graphics processor unit can use a tiled rendering architecture to reduce power consumption or conserve memory bandwidth. As further described above, this rendering method can include a visibility pass that divides a scene into multiple bins and identifies visible triangles in each bin. Thus, in tiled rendering, a full screen can be divided into multiple bins or tiles. The scene can then be rendered multiple times, for example, once or multiple times for each bin.

[0040]

[0047] In graphics rendering aspects, some graphics applications may render to a single target, i.e., a render target, one or more times. For example, in graphics rendering, a frame buffer in system memory may be updated multiple times. A frame buffer may be a portion of memory or random access memory (RAM), for example, containing a bitmap or storage to help store display data for a GPU. A frame buffer may also be a memory buffer containing a complete frame of data. Additionally, a frame buffer may be a logical buffer. In some aspects, updating the frame buffer may be performed in bin or tile rendering, where a surface is divided into multiple bins or tiles, as described above, and each bin or tile may then be rendered separately. Additionally, in tile rendering, a frame buffer may be partitioned into multiple bins or tiles.

[0041]

[0048] In some aspects, such as in a bin or tile rendering architecture as illustrated herein, a frame buffer may have data stored or written to it repeatedly, for example, when rendering from different types of memory. This is sometimes referred to as resolving and unresolving a frame buffer or system memory. For example, when storing or writing to one frame buffer and then switching to another frame buffer, the data or information on the frame buffer may be resolved from GPU internal memory (GMEM) in the GPU to memory in system memory, i.e., double data rate (DDR) RAM or dynamic RAM (DRAM).

[0042]

[0049] In some embodiments, the system memory may also be, for example, a system-on-chip (SoC) memory or another chip-based memory on a device or smartphone for storing data or information. The system memory may also be physical data storage shared by the CPU and / or GPU. In some embodiments, the system memory may be, for example, a DRAM chip on a device or smartphone. Thus, the SoC memory may be a chip-based way of storing data.

[0043]

[0050] In some aspects, GMEM may be on-chip memory in a GPU, which may be implemented by static RAM (SRAM). Furthermore, GMEM may be stored in a device, e.g., a smartphone. As described herein, data or information may be transferred between system memory or DRAM and GMEM in a device, for example. In some aspects, the system memory or DRAM may be in the CPU or GPU. Furthermore, data may be stored in DDR or DRAM. In some aspects, such as in bin or tile rendering, a small portion of memory may be stored in the GPU, e.g., in GMEM. In some cases, storing data in GMEM may utilize a larger processing workload and / or consumed power compared to storing data in a frame buffer or system memory.

[0044]

[0051] Some aspects of image or frame processing may be relevant to different types of applications, for example, extended reality (XR), augmented reality (AR), or virtual reality (VR) applications. In some aspects, XR, AR, or VR applications or systems may be utilized with some devices, for example, mobile devices or smartphones, which may be under strict constraints on power and performance efficiency. To alleviate these constraints, motion estimation may be performed on previously rendered content and / or used to extrapolate frames. For example, instead of rendering frames, previously rendered frames may be used to estimate motion for frames. Motion estimation may allow rendering operations to run at a reduced frame rate. Frame extrapolation may also be utilized to stream remote game rendering, such as to cover intermittent network issues and / or bandwidth constraints.

[0045]

[0052] Although motion estimation may work well sometimes, content may be discontinuous when transitioning between frames, such as from frame to frame. For example, some actions within an application, such as when a user teleports in a gaming application, may result in discontinuous content between frames. In these cases, performing motion estimation may result in inaccurate data based on the discontinuous content between frames, e.g., inaccurate or false motion estimation data. This inaccurate data may result in poor or incomplete frame extrapolation based on motion estimation.

[0046]

[0053] Some types of content may cause the above-mentioned discontinuities between frames. For example, user interface (UI) elements or menus, such as those that pop open or change content in a single frame, may be discontinuous between successive frames. Also, some camera transitions that occur in a single frame, such as the common teleport XR locomotion mechanic, may be discontinuous between frames. Furthermore, a snap turn that rotates the camera a large amount in a single frame may be discontinuous between successive frames. Effects that add instant transparent overlays or brightness changes may also be discontinuous between frames. Furthermore, fast controller movements may be discontinuous between successive frames.

[0047]

[0054] In some cases, motion estimation and / or frame extrapolation applied across these discontinuous frames may produce random or incorrect motion vectors. As previously indicated, the resulting discontinuities in content may be across the entire frame or within certain regions of the frame. Moreover, these random or incorrect motion vectors produced from discontinuous frames may cause strong, unintended distortions in the extrapolated frame or in certain portions of the extrapolated frame.

[0048]

[0055] As described above, motion estimation may specify continuity between frames or some regions of a frame. For example, if there are discontinuities between frames based on frame content or luminance, inaccurate motion estimation may be produced. Based on the above, it may be beneficial to skip or bypass motion estimation for some frames or regions of a frame to avoid incorrect motion vectors and / or unintended distortion. For example, if discontinuities or differences between some frames or regions of a frame are identified, it may be beneficial to skip or bypass performing motion estimation.

[0049]

[0056] Aspects of the present disclosure may skip or bypass performing motion estimation for some frames or regions of frames to avoid incorrect motion vectors and / or unintended distortion. For example, aspects of the present disclosure may identify discontinuities or differences between some frames or regions of frames. Furthermore, aspects of the present disclosure may determine whether the discontinuities or differences between some frames or regions of frames exceed a threshold, and then skip or bypass performing motion estimation if the discontinuities or differences exceed the threshold.

[0050]

[0057] As noted above, motion estimation may produce several estimated vectors even when presented with discontinuous content, which may result in regions of low-quality estimation. Some aspects of the present disclosure may utilize several components, such as lightweight hashes, to perform analysis to identify when content splits between frames or regions of a frame. For example, a frame may be divided into regions, and a hash analysis is performed for each region of the frame. The lightweight hashes of the present disclosure may compare the content of each frame region to determine or identify discontinuities between frame regions.

[0051]

[0058] In some cases, if there is too much change or discontinuity between regions of a frame, a threshold may be triggered that causes motion estimation for the region or frame to be skipped or bypassed. Thus, the results for each frame region may be used to skip vector generation or inform a reliability metric for that particular region. Thus, aspects of the present disclosure may determine or identify discontinuities between frames or frame regions that may not be capable of handling motion estimation, and then skip or bypass motion estimation for those frames or frame regions. Furthermore, the present disclosure may determine or identify those discontinuities between frames or frame regions that may not be capable of handling motion estimation. Furthermore, some aspects of the present disclosure may be utilized with some types of rendering, such as bin rendering or tile rendering.

[0052]

[0059] 4A and 4B illustrate a first frame 400 and a second frame 450, respectively. More specifically, FIG. 4A illustrates the first frame 400, which includes regions 411-418, and FIG. 4B illustrates the second frame 450, which includes regions 451-458. As illustrated in FIGS. 4A and 4B, the first frame 400 and the second frame 450 include multiple regions with different patterns. Similar patterns between corresponding frame regions indicate correlated or continuous content. Similarly, dissimilar patterns between corresponding frame regions indicate uncorrelated or discontinuous content. For example, regions 411, 412, 415, 417, and 418 in the first frame 400 and regions 451, 452, 455, 457, and 458 in the second frame 450 each include a dotted pattern. Therefore, corresponding frame regions, e.g., regions 411 / 451, 412 / 452, 415 / 455, 417 / 457, and 418 / 458, contain correlated or continuous content. Moreover, regions 413, 414, and 416 in first frame 400 contain horizontal patterns, and regions 453, 454, and 456 in second frame 450 contain vertical patterns. Therefore, corresponding frame regions, e.g., regions 413 / 453, 414 / 454, and 416 / 456, contain uncorrelated or discontinuous content.

[0053]

[0060] In some aspects of the present disclosure, scene change detection and resulting motion estimation workload modification may be performed inline, i.e., simultaneously, with the motion estimation process. In some instances, the motion estimation algorithm of the present disclosure may utilize an image pyramid or a partial image pyramid. The image pyramid may be a hierarchy containing a sequence of downscaled versions of an input image. Aspects of the present disclosure may extend the image pyramid to lower resolution levels compared to general motion estimation. Thus, the present disclosure may extend the image pyramid to several resolutions, for example, 8x8 pixels, to perform the above-mentioned hash analysis for each frame region. This hash analysis may produce several discontinuities or hash values ​​for delta calculations that may be compared to a motion estimation threshold.

[0054]

[0061] As described above, aspects of the present disclosure may use an image hash function to detect discontinuities in consecutive frame content, i.e., inter-frame content. This image hash function of the present disclosure may be implemented at a lower resolution image pyramid level, which may be segmented into frame regions. In some aspects, when a region is detected to have a discontinuity, a hash function shader pass may modify a mask that controls which regions of a surface may have motion estimation applied to them. Once the present disclosure identifies a frame region where motion estimation should be performed, a mask or masked motion estimation may be utilized to determine whether a search should be performed at the motion vector level. Thus, once a region of a frame is identified for motion estimation, this information may be provided to the mask or masked motion estimation. If each region or portion of a region is identified or determined to contain discontinuous content, the present disclosure may bypass frame extrapolation.

[0055]

[0062] In some aspects, identifying or determining discontinuous content of a frame or frame region may be performed by a component, such as a GPU or CPU. For example, aspects of the present disclosure may be performed by a GPU during the frame rendering process of an AR / XR / VR application. For example, aspects of the present disclosure may be performed by a GPU or CPU on a host device or a client device, such as a mobile device or smartphone.

[0056]

[0063] The discontinuities between frame regions may be related to differences in content, brightness, and / or noise. Furthermore, the discontinuities between frame regions may correspond to transparent overlays between the regions. For example, a user may be observing an object and activate a UI, which may result in a transparent overlay on a subsequent frame compared to a previous frame. In some aspects, the present disclosure may consider two frames when performing discontinuity identification. In other aspects, the present disclosure may consider more than two frames, e.g., three, four, eight, sixteen frames, etc., when performing discontinuity identification.

[0057]

[0064] FIG. 5 shows a diagram 500 of image or frame processing according to the present disclosure. As shown in FIG. 5, at 502, aspects of the present disclosure may receive several input frames. At 504, aspects of the present disclosure may divide each frame into regions or sub-regions. For each region or sub-region of a frame, the present disclosure may perform steps 512 / 514 / 522 / 524. At 512, aspects of the present disclosure may calculate a hash value between regions or sub-regions of consecutive frames. At 514, aspects of the present disclosure may compare the hash value with a threshold, e.g., a motion estimation threshold. If the hash value does not exceed the threshold, then at 522, aspects of the present disclosure may perform motion estimation for the region or sub-region of the frame. If the hash value exceeds the threshold, then at 524, aspects of the present disclosure may not perform motion estimation for the region or sub-region.

[0058]

[0065] FIG. 6 shows a diagram 600 of image or frame processing according to the present disclosure. As shown in FIG. 6, at 602, embodiments of the present disclosure may utilize a high-resolution image pyramid. At 604, embodiments of the present disclosure may extend the high-resolution image pyramid to a low-resolution pyramid. At 606, embodiments of the present disclosure may utilize a mask, for example, an application motion estimation mask. Based on 604 and 606, embodiments of the present disclosure may perform hash calculations, threshold comparisons, and / or mask modifications at 608. Based on the partial-frame region detection at 602 and / or 608, embodiments of the present disclosure may perform masked motion estimation at 610. Furthermore, at 612, embodiments of the present disclosure may perform frame extrapolation. Based on the full-frame detection at 602 and / or 608, embodiments of the present disclosure may skip or bypass frame extrapolation at 614.

[0059]

[0066] As previously indicated, in some aspects, a lack of scene change detection can result in high distortion on subsequent frames, for example, when a user teleports with an incorrect motion estimate. In contrast, in aspects of the present disclosure that include scene change detection, low-quality motion vectors for successive frames can be identified and removed. This removal of low-quality motion vectors can result in clean extrapolated frames and / or increased accuracy.

[0060]

[0067] 7 illustrates a diagram 700 of image, frame, or graphics processing components. As shown in FIG. 7, the diagram 700 includes a central processing unit (CPU) 710, a graphics processing unit (GPU) 720, and a display 730 including a display processing unit (DPU). The diagram 700 illustrates several components that may be utilized by a client or host device, such as a mobile device or smartphone, in an XR, AR, or VR application. For example, the CPU 710, GPU 720, and display 730 may be utilized to implement the skipping motion estimation techniques described herein.

[0061]

[0068] 4A-7 illustrate examples of the above-described methods and processes for skipping motion estimation. As shown in FIG. 4A-7, in aspects of the present disclosure, the CPU and GPU herein, e.g., CPU 710 and / or GPU 720, may perform several different steps or processes for skipping motion estimation to increase the accuracy of image processing. For example, the CPU and GPU herein may skip motion estimation for a first frame region and a second frame region when the discontinuity between the first region and the second region is greater than or equal to a motion estimation threshold.

[0062]

[0069] A GPU herein, e.g., GPU 720, may receive a second frame, e.g., second frame 450, after a first frame, e.g., first frame 400, is received, where the first frame and the second frame may relate to a plurality of frames. For example, this process may be implemented by step 502 in FIG. 5. The first frame, e.g., first frame 400, and the second frame, e.g., second frame 450, may relate to at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application. Also, the plurality of frames related to the first frame and the second frame may correspond to a video stream.

[0063]

[0070] A GPU herein, e.g., GPU 720, may also compare at least one first region, e.g., region 413, of the first frame with at least one second region, e.g., region 453, of the second frame.

[0064]

[0071] The GPU herein, e.g., GPU 720, may also identify a discontinuity between at least one first region, e.g., region 413, of a first frame and at least one second region, e.g., region 453, of a second frame, wherein the at least one first region corresponds to at least one second region, the first frame includes a plurality of first regions, e.g., regions 411-418, the second frame includes a plurality of second regions, e.g., regions 451-458, and the first frame and the second frame relate to a plurality of frames.

[0065]

[0072] In some aspects, the discontinuity between at least one first region, e.g., region 413, and at least one second region, e.g., region 453, may correspond to non-contiguous content between the at least one first region and the at least one second region. Furthermore, the discontinuity may be associated with a difference in at least one of content, luminance, or noise between the at least one first region, e.g., region 413, and the at least one second region, e.g., region 453. The discontinuity between the at least one first region, e.g., region 413, and the at least one second region, e.g., region 453, may also be associated with multiple motion vectors. Furthermore, the discontinuity between the at least one first region, e.g., region 413, and the at least one second region, e.g., region 453, may be identified by a graphics processing unit (GPU), e.g., GPU 720.

[0066]

[0073] A GPU herein, e.g., GPU 720, may also determine whether a discontinuity between at least one first region, e.g., region 413, and at least one second region, e.g., region 453, is greater than or equal to a motion estimation threshold. For example, this process may be implemented by step 608 in FIG. 6.

[0067]

[0074] A GPU herein, e.g., GPU 720, may also skip motion estimation for at least one first region, e.g., region 413, and at least one second region, e.g., region 453, when a discontinuity between the at least one first region, e.g., region 413, and the at least one second region, e.g., region 453, is greater than or equal to a motion estimation threshold. For example, this process may be implemented by step 524 in FIG. 5. In some cases, at least one of the skipped motion estimations or the identified discontinuity may be associated with at least one of a mask or masked motion estimation, such as in step 610 in FIG. 6. Also, at least one of the skipped motion estimations or the identified discontinuity may be associated with an image pyramid, e.g., in steps 602 and 604 in FIG. 6, or at least one of an image hash function, e.g., in step 608 in FIG. 6.

[0068]

[0075] A GPU herein, e.g., GPU 720, may also perform motion estimation for at least one first region and at least one second region when a discontinuity between at least one first region, e.g., region 413, and at least one second region, e.g., region 453, is less than a motion estimation threshold. For example, this process may be performed by step 522 in FIG. 5. In some aspects, the performed motion estimation may correspond to multiple motion vectors.

[0069]

[0076] The GPU herein, for example, GPU 720, may also construct a third frame based on the performed motion estimation, where the constructed third frame may be associated with a frame extrapolation process, such as in step 612 in FIG. 6.

[0070]

[0077] 8 shows a flowchart 800 of an example method in accordance with one or more techniques of this disclosure. The method may be performed by an apparatus, such as a GPU, a CPU, an image processor, a frame processor, a display processing unit (DPU), or another apparatus for image, frame, or graphics processing.

[0071]

[0078] At 802, the device may receive a second frame after a first frame is received, where the first frame and the second frame may relate to a plurality of frames, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7. As described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, the first frame and the second frame may relate to at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application. Also, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, the plurality of frames related to the first frame and the second frame may correspond to a video stream.

[0072]

[0079] At 804, the device may compare at least one first region of the first frame with at least one second region of the second frame, as described with respect to the examples in Figures 4A, 4B, 5, 6, and 7.

[0073]

[0080] At 806, the apparatus may identify a discontinuity between at least one first region of the first frame and at least one second region of the second frame, as described with respect to the examples in Figures 4A, 4B, 5, 6, and 7, where the at least one first region corresponds to at least one second region, the first frame includes a plurality of first regions, the second frame includes a plurality of second regions, and the first frame and the second frame are related to a plurality of frames.

[0074]

[0081] In some aspects, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, the discontinuity between the at least one first region and the at least one second region may correspond to non-adjacent content between the at least one first region and the at least one second region. Further, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, the discontinuity may be associated with a difference in at least one of content, luminance, or noise between the at least one first region and the at least one second region. As described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, the discontinuity between the at least one first region and the at least one second region may also be associated with multiple motion vectors. Further, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, the discontinuity between the at least one first region and the at least one second region may be identified by a graphics processing unit (GPU).

[0075]

[0082] At 808, the device may determine whether the discontinuity between at least one first region and at least one second region is greater than or equal to a motion estimation threshold, as described with respect to the examples in Figures 4A, 4B, 5, 6, and 7.

[0076]

[0083] At 810, the apparatus may skip motion estimation for at least one first region and at least one second region when a discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7. In some cases, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, at least one of the skipped motion estimation or the identified discontinuity may be associated with at least one of a mask or a masked motion estimation. Also, as described with reference to the examples in Figures 4A, 4B, 5, 6, and 7, at least one of the skipped motion estimation or the identified discontinuity may be associated with at least one of an image pyramid or an image hash function.

[0077]

[0084] At 812, the apparatus may perform motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is less than a motion estimation threshold, as described with respect to the examples in Figures 4A, 4B, 5, 6, and 7. In some aspects, the performed motion estimation may correspond to multiple motion vectors, as described with respect to the examples in Figures 4A, 4B, 5, 6, and 7.

[0078]

[0085] At 814, the device may construct a third frame based on the performed motion estimation, as described with respect to the examples in Figures 4A, 4B, 5, 6, and 7, where the constructed third frame may be related to a frame extrapolation process.

[0079]

[0086] In one configuration, a method or apparatus is provided for image, frame, or graphics processing. The apparatus may be a GPU, a CPU, an image processor, a frame processor, a display processing unit (DPU), or some other processor capable of performing image, frame, or graphics processing. In one aspect, the apparatus may be a processing unit 120 in the device 104, or some other hardware in the device 104 or another device. The apparatus may include means for identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, where the at least one first region corresponds to at least one second region, the first frame includes a plurality of first regions, the second frame includes a plurality of second regions, and the first frame and the second frame relate to a plurality of frames. The apparatus may also include means for determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold. The apparatus may also include means for skipping motion estimation for at least one first region and at least one second region when a discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold. The apparatus may also include means for performing motion estimation for the at least one first region and the at least one second region when a discontinuity between the at least one first region and the at least one second region is less than a motion estimation threshold. The apparatus may also include means for constructing a third frame based on the performed motion estimation, wherein the constructed third frame is associated with a frame extrapolation process. The apparatus may also include means for receiving a second frame after the first frame is received. The apparatus may also include means for comparing at least one first region of the first frame with at least one second region of the second frame.

[0080]

[0087] The subject matter described herein may be implemented to realize one or more benefits or advantages. For example, the described image, frame, or graphics processing techniques may be used by a GPU, CPU, image processor, frame processor, display processing unit (DPU), or any other processor capable of performing image, frame, or graphics processing to implement the motion estimation skip techniques described herein. This may also be achieved at a low cost compared to other image, frame, or graphics processing techniques. Moreover, the image, frame, or graphics processing techniques herein may improve or speed up frame processing or execution. Furthermore, the image, frame, or graphics processing techniques herein may improve resource or data utilization and / or resource efficiency. Furthermore, aspects of the present disclosure may utilize motion estimation skip techniques to increase accuracy, save power, improve processing time, reduce latency, and / or reduce performance overhead.

[0081]

[0088] According to the present disclosure, the term "or" may be interrupted as "and / or" unless the context dictates otherwise. Furthermore, although phrases such as "one or more" or "at least one" may be used for some features disclosed herein and not for other features, the feature for which such phraseology is not used may be construed as having such implied meaning unless the context dictates otherwise.

[0082]

[0089] In one or more examples, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term “processing unit” is used throughout this disclosure, such a processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technique, or other module described herein is implemented in software, the function, processing unit, technique, or other module described herein may be stored on or transmitted via a computer-readable medium as one or more instructions or code. Computer-readable media may include computer data storage media or communication media, including any medium that facilitates transfer of a computer program from one place to another. In this manner, computer-readable media may generally correspond to (1) tangible computer-readable storage media that are non-transitory, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. By way of example and not limitation, such computer-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and Blu-ray discs, where disks typically reproduce data magnetically and discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. A computer program product may include a computer-readable medium.

[0083]

[0090] The code may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), arithmetic logic units (ALUs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor," as used herein, may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0084]

[0091] The techniques of the present disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or sets of ICs, e.g., chipsets. Although various components, modules, or units have been described in this disclosure to highlight functional aspects of devices configured to implement the disclosed techniques, those components, modules, or units do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in any hardware unit or provided by a collection of interoperable hardware units, including one or more processors described above, along with suitable software and / or firmware.

[0085]

[0092] Various examples have been described. These and other examples are within the scope of the following claims.

[0086]

[0093] The following aspects are exemplary only and can be combined with, but not limited to, other aspects or teachings described herein.

[0087]

[0094] Aspect 1 is a method of image processing, including: identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, the at least one first region corresponding to at least one second region, the first frame including a plurality of first regions, the second frame including a plurality of second regions, the first frame and the second frame relating to the plurality of frames; determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; and skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold.

[0088]

[0095] Example 2 is the method of example 1, wherein the discontinuity between the at least one first region and the at least one second region corresponds to non-adjacent content between the at least one first region and the at least one second region.

[0089]

[0096] Example 3 is a method according to any of Examples 1 and 2, wherein the discontinuity is associated with a difference in at least one of content, brightness, or noise between the at least one first region and the at least one second region.

[0090]

[0097] Example 4 is the method of any of Examples 1 to 3, wherein at least one of the skipped motion estimates or the identified discontinuities is associated with at least one of the mask or the masked motion estimates.

[0091]

[0098] Example 5 is the method of any of Examples 1 to 4, wherein at least one of the skipped motion estimation or the identified discontinuity is associated with at least one of an image pyramid or an image hash function.

[0092]

[0099] Example 6 is a method of any of Examples 1 to 5, further comprising: performing motion estimation for the at least one first region and the at least one second region when a discontinuity between the at least one first region and the at least one second region is less than a motion estimation threshold.

[0093]

[0100] Example 7 is a method described in any of Examples 1 to 6, further comprising constructing a third frame based on the performed motion estimation, wherein the constructed third frame is associated with a frame extrapolation process.

[0094]

[0101] Example 8 is the method of any of Examples 1 to 7, wherein the performed motion estimation corresponds to a plurality of motion vectors.

[0095]

[0102] Example 9 is the method of any of Examples 1 to 8, further comprising receiving a second frame after the first frame is received.

[0096]

[0103] Example 10 is the method of any of Examples 1 to 9, further comprising comparing at least one first region of the first frame with at least one second region of the second frame.

[0097]

[0104] Example 11 is the method of any of Examples 1 to 10, wherein the discontinuity between the at least one first region and the at least one second region is associated with a plurality of motion vectors.

[0098]

[0105] Example 12 is a method according to any of Examples 1 to 11, wherein the first frame and the second frame are associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application.

[0099]

[0106] Example 13 is the method of any of Examples 1 to 12, wherein a discontinuity between at least one first region and at least one second region is identified by a graphics processing unit (GPU).

[0100]

[0107] Example 14 is the method of any of Examples 1 to 13, wherein the plurality of frames related to the first frame and the second frame corresponds to a video stream.

[0101]

[0108] Aspect 15 is an apparatus for image processing, comprising means for implementing the method according to any one of aspects 1 to 14.

[0102]

[0109] Aspect 16 is an apparatus for image processing, including at least one processor coupled to a memory and configured to implement a method according to any of aspects 1-14.

[0103]

[0110] Aspect 17 is a computer-readable medium storing computer-executable code, which when executed by a processor causes the processor to implement a method according to any of aspects 1-14. The inventions described in the claims of the present application as originally filed are set forth below. [C1] 1. A method of image processing, comprising: identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, wherein the at least one first region corresponds to the at least one second region, the first frame includes a plurality of first regions, the second frame includes a plurality of second regions, and the first frame and the second frame relate to a plurality of frames; determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; and skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold. A method comprising: [C2] The method of C1, wherein the discontinuity between the at least one first region and the at least one second region corresponds to non-adjacent content between the at least one first region and the at least one second region. [C3] The method of C1, wherein the discontinuity is related to a difference in at least one of content, brightness, or noise between the at least one first region and the at least one second region. [C4] The method of C1, wherein at least one of the skipped motion estimate or the identified discontinuity is associated with at least one of a mask or a masked motion estimate. [C5] The method of C1, wherein at least one of the skipped motion estimation or the identified discontinuity is associated with at least one of an image pyramid or an image hash function. [C6] performing motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is less than the motion estimation threshold. The method of C1, further comprising: [C7] and constructing a third frame based on the performed motion estimation, wherein the constructed third frame is associated with a frame extrapolation process. The method described in C6. [C8] The method of C6, wherein the performed motion estimation corresponds to a plurality of motion vectors. [C9] receiving the second frame after the first frame is received; The method of C1, further comprising: [C10] comparing the at least one first region of the first frame with the at least one second region of the second frame; The method of C1, further comprising: [C11] The method of C1, wherein the discontinuity between the at least one first region and the at least one second region is associated with a plurality of motion vectors. [C12] The method of C1, wherein the first frame and the second frame are associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application. [C13] The method of C1, wherein the discontinuity between the at least one first region and the at least one second region is identified by a graphics processing unit (GPU). [C14] The method of C1, wherein the plurality of frames related to the first frame and the second frame correspond to a video stream. [C15] 1. An apparatus for image processing, comprising: Memory and at least one processor coupled to the memory; wherein the at least one processor: identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, wherein the at least one first region corresponds to the at least one second region, the first frame includes a plurality of first regions, the second frame includes a plurality of second regions, and the first frame and the second frame relate to a plurality of frames; determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold; An apparatus configured to: [C16] The device of C15, wherein the discontinuity between the at least one first region and the at least one second region corresponds to non-adjacent content between the at least one first region and the at least one second region. [C17] The apparatus of C15, wherein the discontinuity is associated with a difference in at least one of content, brightness, or noise between the at least one first region and the at least one second region. [C18] The apparatus of C15, wherein at least one of the skipped motion estimate or the identified discontinuity is associated with at least one of a mask or a masked motion estimate. [C19] The apparatus of C15, wherein at least one of the skipped motion estimation or the identified discontinuity is associated with at least one of an image pyramid or an image hash function. [C20] The at least one processor: performing motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is less than the motion estimation threshold. The apparatus of C15, further configured to: [C21] the at least one processor: and further configured to construct a third frame based on the performed motion estimation, wherein the constructed third frame is associated with a frame extrapolation process. The device described in C20. [C22] The apparatus of C20, wherein the performed motion estimation corresponds to a plurality of motion vectors. [C23] The at least one processor: receiving the second frame after the first frame is received; The apparatus of C15, further configured to: [C24] the at least one processor: comparing the at least one first region of the first frame with the at least one second region of the second frame; The apparatus of C15, further configured to: [C25] The apparatus of C15, wherein the discontinuity between the at least one first region and the at least one second region is associated with a plurality of motion vectors. [C26] The apparatus of C15, wherein the first frame and the second frame are associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application. [C27] The apparatus of C15, wherein the discontinuity between the at least one first region and the at least one second region is identified by a graphics processing unit (GPU). [C28] The apparatus of C15, wherein the plurality of frames related to the first frame and the second frame correspond to a video stream. [C29] 1. An apparatus for image processing, comprising: means for identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, wherein the at least one first region corresponds to the at least one second region, the first frame includes a plurality of first regions, the second frame includes a plurality of second regions, and the first frame and the second frame relate to a plurality of frames; means for determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; means for skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold; An apparatus comprising: [C30] 30. The apparatus of claim 29, wherein the discontinuity between the at least one first region and the at least one second region corresponds to non-adjacent content between the at least one first region and the at least one second region. [C31] The apparatus of C29, wherein the discontinuity is associated with a difference in at least one of content, brightness, or noise between the at least one first region and the at least one second region. [C32] The apparatus of C29, wherein at least one of the skipped motion estimate or the identified discontinuity is associated with at least one of a mask or a masked motion estimate. [C33] The apparatus of C29, wherein at least one of the skipped motion estimation or the identified discontinuity is associated with at least one of an image pyramid or an image hash function. [C34] means for performing motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is less than the motion estimation threshold. The apparatus of C29, further comprising: [C35] The apparatus of C34, further comprising means for constructing a third frame based on the performed motion estimation, wherein the constructed third frame is associated with a frame extrapolation process. [C36] The apparatus of C34, wherein the performed motion estimation corresponds to a plurality of motion vectors. [C37] means for receiving the second frame after the first frame is received. The apparatus of C29, further comprising: [C38] means for comparing the at least one first region of the first frame with the at least one second region of the second frame; The apparatus of C29, further comprising: [C39] The apparatus of C29, wherein the discontinuity between the at least one first region and the at least one second region is associated with a plurality of motion vectors. [C40] The apparatus of C29, wherein the first frame and the second frame are associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application. [C41] The apparatus of C29, wherein the discontinuity between the at least one first region and the at least one second region is identified by a graphics processing unit (GPU). [C42] The apparatus of C29, wherein the plurality of frames related to the first frame and the second frame correspond to a video stream. [C43] 1. A computer-readable medium storing computer-executable code for image processing, the code, when executed by a processor, causing the processor to: identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, wherein the at least one first region corresponds to the at least one second region, the first frame includes a plurality of first regions, the second frame includes a plurality of second regions, and the first frame and the second frame relate to a plurality of frames; determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold; A computer-readable medium for causing

Claims

1. 1. A method of image processing, comprising: identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, wherein the at least one first region corresponds to the at least one second region, the first frame including a plurality of first regions, the second frame including a plurality of second regions, and the first frame and the second frame relating to a plurality of frames, wherein the discontinuity is identified by applying an image hash function to the first region and the second region to compare the first region and the second region, the image hash function being performed at a lower resolution image pyramid level that may be segmented into frame regions; determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold; A method comprising:

2. 2. The method of claim 1 , wherein the discontinuity between the at least one first region and the at least one second region corresponds to non-adjacent content between the at least one first region and the at least one second region.

3. The method of claim 1 , wherein the discontinuity is related to a difference in at least one of content, brightness, or noise between the at least one first region and the at least one second region.

4. 2. The method of claim 1 , wherein at least one of the skipped motion estimation or the identified discontinuity is associated with at least one of a mask or a masked motion estimation that controls to which of the at least one first region and the at least one second region motion estimation is applied.

5. 2. The method of claim 1 , wherein at least one of the skipped motion estimation or the identified discontinuity is associated with at least one of an image pyramid comprising a layer containing a sequence of downscaled versions of the first frame and the second frame.

6. performing motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is less than the motion estimation threshold. The method of claim 1 further comprising:

7. and constructing a third frame based on the performed motion estimation, wherein the constructed third frame is associated with a frame extrapolation process. The method of claim 6.

8. The method of claim 6 , wherein the performed motion estimation corresponds to a plurality of motion vectors.

9. receiving the second frame after the first frame is received; The method of claim 1 further comprising:

10. comparing the at least one first region of the first frame with the at least one second region of the second frame; and / or the discontinuity between the at least one first region and the at least one second region is associated with a plurality of motion vectors; and / or the first frame and the second frame are associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application; and / or the discontinuity between the at least one first region and the at least one second region is identified by a graphics processing unit (GPU); and / or The method of claim 1 , wherein the plurality of frames associated with the first frame and the second frame correspond to a video stream.

11. 1. An apparatus for image processing, comprising: Memory and at least one processor coupled to the memory; wherein the at least one processor identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, wherein the at least one first region corresponds to the at least one second region, the first frame including a plurality of first regions, the second frame including a plurality of second regions, and the first frame and the second frame relating to a plurality of frames, wherein the discontinuity is identified by applying an image hash function to the first region and the second region to compare the first region and the second region, the image hash function being performed at a lower resolution image pyramid level that may be segmented into frame regions; determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold; An apparatus configured to:

12. the discontinuity between the at least one first region and the at least one second region corresponds to non-adjacent content between the at least one first region and the at least one second region; and / or the discontinuity is related to a difference in at least one of content, brightness, or noise between the at least one first region and the at least one second region; and / or At least one of the skipped motion estimation or the identified discontinuity is associated with at least one of a mask or a masked motion estimation that controls to which of the at least one first region and the at least one second region motion estimation is applied; and / or At least one of the skipped motion estimation or the identified discontinuity is associated with at least one of an image pyramid comprising a layer containing a sequence of downscaled versions of the first frame and the second frame; and / or The at least one processor performing motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is less than the motion estimation threshold. The apparatus of claim 11 , further configured to:

13. the at least one processor:

13. The apparatus of claim 12, further configured to construct a third frame based on the performed motion estimation, wherein the constructed third frame is related to a frame extrapolation process and / or the performed motion estimation corresponds to a plurality of motion vectors.

14. The at least one processor receiving the second frame after the first frame is received; and / or the at least one processor: comparing the at least one first region of the first frame with the at least one second region of the second frame; and / or the discontinuity between the at least one first region and the at least one second region is associated with a plurality of motion vectors; and / or the first frame and the second frame are associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application; and / or the discontinuity between the at least one first region and the at least one second region is identified by a graphics processing unit (GPU); and / or The apparatus of claim 11 , wherein the plurality of frames associated with the first frame and the second frame correspond to a video stream.

15. 1. A computer-readable medium storing computer-executable code for image processing, the code, when executed by a processor, causing the processor to: identifying a discontinuity between at least one first region of a first frame and at least one second region of a second frame, wherein the at least one first region corresponds to the at least one second region, the first frame including a plurality of first regions, the second frame including a plurality of second regions, and the first frame and the second frame relating to a plurality of frames, wherein the discontinuity is identified by applying an image hash function to the first region and the second region to compare the first region and the second region, the image hash function being performed at a lower resolution image pyramid level that may be segmented into frame regions; determining whether the discontinuity between the at least one first region and the at least one second region is greater than or equal to a motion estimation threshold; skipping motion estimation for the at least one first region and the at least one second region when the discontinuity between the at least one first region and the at least one second region is greater than or equal to the motion estimation threshold; A computer-readable medium for causing

Images