Motion Estimation via Input Perturbations

Abstract

The present disclosure relates to methods and apparatus for motion estimation, which may include a GPU. In one aspect, the GPU may generate at least one first motion vector in a first subset of frames, the first motion vector providing a first motion estimate for image data in the first subset of frames. The GPU may also perturb the image data. Furthermore, the GPU may generate at least one second motion vector based on the perturbed image data, the second motion vector providing a second motion estimate for the image data. Furthermore, the GPU may compare the first motion vector with the second motion vector. Furthermore, the GPU may determine at least one third motion vector for use in motion estimation of the image data based on the comparison between the first motion vector and the second motion vector.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional patent application No. 16 / 215,547, filed on December 10, 2018, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to processing systems and, more particularly, to one or more techniques for graphics processing in processing systems. Background Art

[0004] Computing devices typically utilize a graphics processing unit (GPU) to accelerate the rendering of graphics data for display. For example, such computing devices may include computer workstations, mobile phones such as so-called smart phones, embedded systems, personal computers, tablet computers, and video game consoles. The GPU executes a graphics processing pipeline comprising multiple processing stages that operate together to execute graphics processing commands and output frames. A central processing unit (CPU) can control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern CPUs are typically capable of executing multiple applications concurrently, and each application may need to utilize a GPU during execution. Devices that provide content for visual presentation on a display typically include a graphics processing unit (GPU).

[0005] Typically, a device's GPU is configured to execute every process in the graphics processing pipeline. However, with the advent of wireless communications and content streaming (e.g., gaming content or any other content rendered using a GPU), a need has developed for distributed graphics processing. For example, a need has developed to offload processing performed by a GPU on a first device (e.g., a client device such as a game console, virtual reality device, or any other device) to a second device (e.g., a server, such as one hosting a mobile game). Summary of the Invention

[0006] The following presents a simplified overview of one or more aspects in order to provide a basic understanding of these aspects. This overview is not an extensive overview of all contemplated aspects and is neither intended to identify key or critical elements of all aspects nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0007] In disclosed aspects, a method, a computer-readable medium, and a first apparatus are provided. The apparatus may be a GPU. In one aspect, the GPU may generate at least one first motion vector in a first subset of a frame, the first motion vector providing a first motion estimate for image data in the first subset of the frame. The GPU may also perturb the image data in the first subset of the frame. Furthermore, the GPU may generate at least one second motion vector based on the perturbed image data, the second motion vector providing a second motion estimate for the image data in the first subset of the frame. Furthermore, the GPU may compare the first motion vector with the second motion vector. Furthermore, the GPU may determine at least one third motion vector for use in the motion estimate of the image data in the first subset of the frame based on the comparison between the first motion vector and the second motion vector.

[0008] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an example content generation and coding system according to techniques of this disclosure.

[0010] Figure 2 An example of motion estimation according to the present disclosure is shown.

[0011] Figure 3A and Figure 3B An example of an image on which motion estimation is performed according to the present disclosure is shown.

[0012] Figure 4A Another example of motion estimation according to the present disclosure is shown.

[0013] Figure 4B Another example of motion estimation according to the present disclosure is shown.

[0014] Figure 5A and Figure 5B Another example of motion estimation according to the present disclosure is shown.

[0015] Figure 6 An example flow chart illustrating an example method in accordance with one or more disclosed techniques is shown. DETAILED DESCRIPTION

[0016] Various aspects of the systems, devices, computer program products, and methods are described more fully below with reference to the accompanying drawings. However, the present disclosure can be embodied in many different forms and should not be construed as limited to any specific structure or function presented throughout the present disclosure. On the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. Based on the teachings herein, those skilled in the art will understand that the scope of the present disclosure is intended to cover any aspect of the systems, devices, computer program products, and methods disclosed herein, whether independent of or in combination with other aspects of the present disclosure. For example, any number of aspects set forth herein can be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such a device or method that uses other structures, functions, or structures and functions in addition to the various aspects of the disclosure set forth herein or in addition to the various aspects of the disclosure set forth herein to practice. Any aspect disclosed herein may be embodied by one or more elements of the claims.

[0017] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the present disclosure. Although some potential benefits and advantages of aspects of the present disclosure are provided, the scope of the present disclosure is not intended to be limited to a particular benefit, use, or objective. On the contrary, aspects of the present disclosure are intended to be broadly applicable to different wireless technologies, system configurations, networks, and transmission protocols, some of which are illustrated by way of example in the figures and in the following description. The detailed description and accompanying drawings are merely illustrative of the present disclosure and are not intended to be limiting. The scope of the present disclosure is defined by the appended claims and their equivalents.

[0018] Several aspects are presented with reference to various devices and methods. These aspects and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0019] For example, an element or any part of an element or any combination of elements can be implemented as a "processing system" including one or more processors (which may also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on chip (SoCs), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gate logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described throughout this disclosure. One or more processors in a processing system can execute software. Software should be broadly interpreted as an average instruction, instruction set, code, code segment, program code, program, subroutine, software component, application, software application, software package, routine, subroutine, object, executable file, execution thread, process, function, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or other. The term application can refer to software. As described herein, one or more technologies may refer to an application (i.e., software) configured to perform one or more functions. In this example, the application may be stored in a memory (e.g., on-chip memory of a processor, system memory, or any other memory). The hardware described herein (such as a processor) may be configured to execute the application. For example, the application may be described as including code that, when executed by hardware, causes the hardware to perform one or more technologies described herein. As an example, the hardware may access code from the memory and execute the code accessed from the processor to perform one or more technologies described herein. In some examples, components are identified in this disclosure. In such examples, a component may be hardware, software, or a combination thereof. A component may be a separate component or a subcomponent of a single component.

[0020] Therefore, in one or more examples described herein, the functions described can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on one or more instructions or codes on a computer-readable medium, or encoded as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available medium that can be accessed by a computer. For example, and not limitation, such computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, a combination of computer-readable media of the above types, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0021] In general, this disclosure describes techniques for enabling a distributed graphics processing pipeline across multiple devices, improving the encoding and decoding of graphical content, and / or reducing the load on a processing unit (i.e., any processor configured to perform one or more of the techniques described herein, such as a graphics processing unit (GPU)). For example, this disclosure describes techniques for graphics processing in a communication system. Other example benefits are described throughout this disclosure.

[0022] As used herein, the term "codec" may refer generally to an encoder and / or a decoder. For example, a reference to a "content codec" may include a reference to a content encoder and / or a content decoder. Similarly, as used herein, the term "codec" may refer generally to encoding and / or decoding. As used herein, the terms "encoding" and "compression" may be used interchangeably. Similarly, the terms "decoding" and "decompression" may be used interchangeably.

[0023] As used herein, instances of the term "content" may refer to the terms "video," "graphics content," "image," and vice versa. This is true regardless of whether these terms are used as adjectives, nouns, or other parts of speech. For example, a reference to a "content encoder" may include a reference to a "video codec," a "graphics content codec," or an "image codec"; a reference to a "video codec," a "graphics content codec," or an "image codec" may include a reference to a "content codec." As another example, a reference to a processing unit that provides content to a content codec may include a reference to a processing unit that provides graphics content to a video codec. In some examples, as used herein, the term "graphics content" may refer to content generated by one or more processes in a graphics processing pipeline. In some examples, as used herein, the term "graphics content" may refer to content generated by a processing unit configured to perform graphics processing. In some examples, as used herein, the term "graphics content" may refer to content generated by a graphics processing unit.

[0024] As used herein, instances of the term "content" may refer to either graphical content or display content. In some examples, as used herein, the term "graphics content" may refer to content generated by a processing unit configured to perform graphics processing. For example, the term "graphics content" may refer to content generated by one or more processes in a graphics processing pipeline. In some examples, as used herein, the term "graphics content" may refer to content generated by a graphics processing unit. In some examples, as used herein, the term "display content" may refer to content generated by a processing unit configured to perform display processing. In some examples, as used herein, the term "display content" may refer to content generated by a display processing unit. Graphics content may be processed into display content. For example, a graphics processing unit may output graphical content, such as a frame, to a buffer (which may be referred to as a frame buffer). A display processing unit may read the graphical content (e.g., one or more frames) from the buffer and perform one or more display processing techniques on it to generate display content. For example, a display processing unit may be configured to combine one or more rendering layers to generate a frame. As another example, a display processing unit may be configured to combine, blend, or otherwise combine two or more layers to form a single frame. The display processing unit can be configured to perform scaling (e.g., zooming in or out) on a frame. In some examples, a frame can refer to a layer. In other examples, a frame can refer to two or more layers that have been blended together to form a frame (i.e., a frame includes two or more layers, and a frame including two or more layers can then be blended).

[0025] As referenced herein, a first component (e.g., a processing unit) can provide content, such as graphical content, to a second component (e.g., a content codec). In some examples, the first component can provide content to the second component by storing the content in a memory accessible to the second component. In such examples, the second component can be configured to read content stored in the memory by the first component. In other examples, the first component can provide content to the second component without any intermediate components (e.g., without a memory or another component). In such examples, the first component can be described as providing content directly to the second component. For example, the first component can output content to the second component, and the second component can be configured to store the content received from the first component in a memory, such as a buffer.

[0026] Figure 1is a block diagram illustrating an example content generation and encoding system 100 configured to implement one or more techniques of this disclosure. Content generation and encoding system 100 includes a source device 102 and a destination device 104. In accordance with the techniques described herein, source device 102 can be configured to encode graphical content generated by processing unit 106 using content encoder 108 prior to transmission to destination device 104. Content encoder 108 can be configured to output a bitstream having a bit rate. Processing unit 106 can be configured to control and / or influence the bit rate of content encoder 108 based on how processing unit 106 generates the graphical content.

[0027] Source device 102 may include one or more components (or circuits) for performing the various functions described herein. Destination device 104 may include one or more components (or circuits) for performing the various functions described herein. In some examples, one or more components in source device 102 may be components of a system on a chip (SOC). Similarly, in some examples, one or more components in destination device 104 may be components of a SOC.

[0028] Source device 102 may include one or more components configured to perform one or more techniques of this disclosure. In the example shown, source device 102 may include a processing unit 106, a content encoder 108, a system memory 110, and a communication interface 112. Processing unit 106 may include internal memory 109. Processing unit 106 may be configured to perform graphics processing (e.g., in graphics processing pipeline 107-1). Content encoder 108 may include internal memory 111.

[0029] The processing unit 106 and the content encoder 108 may access memory external to the processing unit 106 and the content encoder 108, such as the system memory 110. For example, the processing unit 106 and the content encoder 108 may be configured to read from and / or write to the external memory, such as the system memory 110. The processing unit 106 and the content encoder 108 may be communicatively coupled to the system memory 110 via a bus. In some examples, the processing unit 106 and the content encoder 108 may be communicatively coupled to each other via a bus or a different connection.

[0030] The content encoder 108 can be configured to receive graphics content from any source, such as system memory 110 and / or processing unit 106. The system memory 110 can be configured to store the graphics content generated by the processing unit 106. For example, the processing unit 106 can be configured to store the graphics content in the system memory 110. The content encoder 108 can be configured to receive the graphics content in the form of pixel data (e.g., from the system memory 110 and / or processing unit 106). Stated otherwise, the content encoder 108 can be configured to receive pixel data for the graphics content generated by the processing unit 106. For example, the content encoder 108 can be configured to receive a value for each component (e.g., each color component) of one or more pixels in the graphics content. As an example, a pixel in a red (R), green (G), blue (B) (RGB) color space can include a first value for the red component, a second value for the green component, and a third value for the blue component.

[0031] Internal memory 109, system memory 110, and / or internal memory 111 may include one or more volatile or non-volatile memory or storage devices. In some examples, internal memory 109, system memory 110, and / or internal memory 111 may include random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media, optical storage media, or any other type of memory.

[0032] According to some examples, internal memory 109, system memory 110, and / or internal memory 111 may be non-transitory storage media. The term "non-transitory" may indicate that the storage media is not embodied in a carrier wave or propagated signal. However, the term "non-transitory" should not be interpreted to mean that internal memory 109, system memory 110, and / or internal memory 111 are non-removable or that their contents are static. As an example, system memory 110 may be removable from source device 102 and moved to another device. As another example, system memory 110 may not be removable from source device 102.

[0033] Processing unit 106 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, processing unit 106 may be integrated into the motherboard of source device 102. In some examples, processing unit 106 may reside on a graphics card installed in a port on the motherboard of source device 102, or may be otherwise incorporated into a peripheral device configured to interoperate with source device 102.

[0034] The processing unit 106 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the processing unit 106 may store the instructions of the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 109) and may use one or more processors to execute the instructions in hardware to perform the technology of the present disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0035] The content encoder 108 can be any processing unit configured to perform content encoding. In some examples, the content encoder 108 can be integrated into the mainboard of the source device 102. The content encoder 108 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the content encoder 108 can store the instructions of the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 111), and can use one or more processors to execute the instructions in hardware to perform the technology of the present disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be considered to be one or more processors.

[0036] Communication interface 112 may include a receiver 114 and a transmitter 116. Receiver 114 may be configured to perform any of the receiving functions described herein with respect to source device 102. For example, receiver 114 may be configured to receive information from destination device 104, which may include a request for content. In some examples, in response to receiving the request for content, source device 102 may be configured to perform one or more of the techniques described herein, such as generating or otherwise producing graphical content for transmission to destination device 104. Transmitter 116 may be configured to perform any of the transmitting functions described herein with respect to source device 102. For example, transmitter 116 may be configured to transmit encoded content, such as encoded graphical content generated by processing unit 106 and content encoder 108 (i.e., graphical content generated by processing unit 106, which receives the graphical content as input and generates or otherwise produces the encoded graphical content), to destination device 104. Receiver 114 and transmitter 116 may be combined into transceiver 118. In such examples, transceiver 118 may be configured to perform any of the receiving functions and / or transmitting functions described herein with respect to source device 102 .

[0037] Destination 1104 may include one or more components configured to perform one or more techniques of this disclosure. In the example shown, the destination device may include a processing unit 120, a content decoder 122, a system memory 124, a communication interface 126, and one or more displays 131. References to display 131 may refer to one or more displays 131. For example, display 131 may include a single display and multiple displays. Display 131 may include a first display and a second display. The first display may be a left-eye display and the second display may be a right-eye display. In some examples, the first display and the second display may receive different frames for presentation thereon. In other examples, the first display and the second display may receive the same frame for presentation thereon.

[0038] Processing unit 120 may include internal memory 121. Processing unit 120 may be configured (e.g., in graphics processing pipeline 107-2) to perform graphics processing. Content decoder 122 may include internal memory 123. In some examples, destination device 104 may include a display processor (such as display processor 127) to perform one or more display processing techniques on one or more frames generated by processing unit 120 prior to presentation by one or more displays 131. Display processor 127 may be configured to perform display processing. For example, display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by processing unit 120. One or more displays 131 may be configured to display content generated using decoded content. For example, display processor 127 may be configured to process one or more frames generated by processing unit 120, where the one or more frames were generated by processing unit 120 using decoded content derived from encoded content received from source device 102. Display processor 127 may be configured to perform display processing on the one or more frames generated by processing unit 120. One or more displays 131 may be configured to display or otherwise present frames processed by display processor 127. In some examples, the one or more displays may include one or more of a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0039] The processing unit 120 and the content decoder 122 can access memory external to the processing unit 120 and the content decoder 122, such as the system memory 124. For example, the processing unit 120 and the content decoder 122 can be configured to read from and / or write to the external memory, such as the system memory 124. The processing unit 120 and the content decoder 122 can be communicatively coupled to the system memory 124 via a bus. In some examples, the processing unit 120 and the content decoder 122 can be communicatively coupled to each other via a bus or a different connection.

[0040] Content decoder 122 may be configured to receive graphics content from any source, such as system memory 124 and / or communication interface 126. System memory 124 may be configured to store received encoded graphics content, such as encoded graphics content received from source device 102. Content decoder 122 may be configured to receive the encoded graphics content in the form of encoded pixel data (e.g., from system memory 124 and / or communication interface 126). Content decoder 122 may be configured to decode the encoded graphics content.

[0041] Internal memory 121, system memory 124, and / or internal memory 123 may include one or more volatile or non-volatile memory or storage devices. In some examples, internal memory 121, system memory 124, and / or internal memory 123 may include random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media, optical storage media, or any other type of memory.

[0042] According to some examples, internal memory 121, system memory 124, and / or internal memory 123 may be non-transitory storage media. The term "non-transitory" may indicate that the storage media is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that internal memory 121, system memory 124, and / or internal memory 123 are non-removable or that their contents are static. As an example, system memory 124 may be removable from destination device 104 and moved to another device. As another example, system memory 124 may not be removable from destination device 104.

[0043] Processing unit 120 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that may be configured to perform graphics processing. In some examples, processing unit 120 may be integrated into the motherboard of destination device 104. In some examples, processing unit 120 may reside on a graphics card installed in a port in the motherboard of destination device 104, or may otherwise be incorporated into a peripheral device configured to interoperate with destination device 104.

[0044] The processing unit 120 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the processing unit 120 may store the instructions of the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 121) and may use one or more processors to execute the instructions in hardware to perform the technology of the present disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0045] The content decoder 122 can be any processing unit configured to perform content decoding. In some examples, the content decoder 122 can be integrated into the motherboard of the destination device 104. The content decoder 122 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the content decoder 122 may store the instructions of the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 123) and may execute the instructions in hardware using one or more processors to perform the technology of the present disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0046] The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any of the receiving functions described herein with respect to the destination device 104. For example, the receiver 128 may be configured to receive information from the source device 102, which may include encoded content (e.g., encoded graphics content generated or otherwise produced by the processing unit 106 and the content encoder 108 of the source device 102 (i.e., the graphics content is generated by the processing unit 106 and the content encoder 108 receives the graphics content as input to produce or otherwise generate the encoded graphics content)). As another example, the receiver 114 may be configured to receive position information from the destination device 104, which may be encoded or non-encoded (i.e., unencoded). Furthermore, the receiver 128 may be configured to receive position information from the source device 102. In some examples, the destination device 104 may be configured to decode the encoded graphics content received from the source device 102 according to the techniques described herein. For example, content decoder 122 may be configured to decode encoded graphics content to produce or otherwise generate decoded graphics content. Processing unit 120 may be configured to use the decoded graphics content to produce or otherwise generate one or more frames for presentation on one or more displays 131. Transmitter 130 may be configured to perform any of the transmitting functions described herein with respect to destination device 104. For example, transmitter 130 may be configured to transmit information to source device 102, which may include a request for content. Receiver 128 and transmitter 130 may be combined into transceiver 132. In such an example, transceiver 132 may be configured to perform any of the receiving functions and / or transmitting functions described herein with respect to destination device 104.

[0047] The content encoder 108 and the content decoder 122 of the content generation and encoding and decoding system 100 represent examples of computing components (e.g., processing units) that can be configured to perform one or more techniques for encoding and decoding content, respectively, according to various examples described in this disclosure. In some examples, the content encoder 108 and the content decoder 122 can be configured to operate according to a content encoding and decoding standard, such as a video encoding and decoding standard, a display stream compression standard, or an image compression standard.

[0048] like Figure 1 As shown, source device 102 may be configured to generate encoded content. Thus, source device 102 may be referred to as a content encoding device or content encoding apparatus. Destination device 104 may be configured to decode the encoded content generated by source device 102. Thus, destination device 104 may be referred to as a content decoding device or content decoding apparatus. In some examples, as shown, source device 102 and destination device 104 may be separate devices. In other examples, source device 102 and destination device 104 may be on the same computing device or part of the same computing device. In either example, the graphics processing pipeline may be distributed between the two devices. For example, a single graphics processing pipeline may include multiple graphics processes. Graphics processing pipeline 107-1 may include one or more of the multiple graphics processes. Similarly, graphics processing pipeline 107-2 may include one or more of the multiple graphics processes. In this regard, graphics processing pipeline 107-1 connected to or otherwise followed by graphics processing pipeline 107-2 may result in a complete graphics processing pipeline. Described otherwise, graphics processing pipeline 107 - 1 may be a partial graphics processing pipeline and graphics processing pipeline 107 - 2 may be a partial graphics processing pipeline, which when combined, result in a distributed graphics processing pipeline.

[0049] Refer again Figure 1In certain aspects, graphics processing pipeline 107-2 may include a generation component configured to generate at least one first motion vector in a first subset of a frame, the first motion vector providing a first motion estimate for image data in the first subset of the frame. Graphics processing pipeline 107-2 may also include a perturbation component configured to perturb the image data in the first subset of the frame. Furthermore, the generation component may be configured to generate at least one second motion vector based on the perturbed image data in the first subset of the frame, the second motion vector providing a second motion estimate for the image data in the first subset of the frame. Graphics processing pipeline 107-2 may also include a comparison component configured to compare the first motion vector with the second motion vector. Furthermore, graphics processing pipeline 107-2 may include a determination component 198 configured to determine at least one third motion vector for use in the motion estimate of the image data in the first subset of the frame based on the comparison between the first motion vector and the second motion vector. In some examples, by distributing the graphics processing pipeline between source device 102 and destination device 104, the destination device may be able to render graphical content that would otherwise be incapable of rendering and, therefore, unavailable for presentation. Other example benefits are described throughout this disclosure.

[0050] As described herein, a device such as source device 102 and / or destination device 104 may refer to any device, apparatus, or system configured to perform one or more of the techniques described herein. For example, a device may be a server, a base station, a user device, a client device, a station, an access point, a computer (e.g., a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation, or a mainframe computer), an end product, an apparatus, a phone, a smartphone, a server, a video game platform or console, a handheld device (e.g., a portable video game device or a personal digital assistant (PDA)), a wearable computing device (e.g., a smartwatch, an augmented reality device, or a virtual reality device), a non-wearable device, an augmented reality device, a virtual reality device, a display (e.g., a display device), a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, an in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more of the techniques described herein.

[0051] Source device 102 can be configured to communicate with destination device 104. For example, destination device 104 can be configured to receive encoded content from source device 102. In some examples, the communicative coupling between source device 102 and destination device 104 is shown as link 134. Link 134 can include any type of medium or device capable of moving encoded content from source device 102 to destination device 104.

[0052] exist Figure 1In an example, link 134 may include a communication medium to enable source device 102 to send encoded content to destination device 104 in real time. The encoded content may be modulated according to a communication standard (such as a wireless communication standard) and sent to destination device 104. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network (such as a local area network, a wide area network, or a global network such as the Internet). The communication medium may include a router, a switch, a base station, or any other device that can facilitate communication from source device 102 to destination device 104. In other examples, link 134 may be a point-to-point connection between source device 102 and destination device 104, such as a wired or wireless display link connection (e.g., an HDMI link, a display port link, a MIPI DSI link, or another link through which encoded content can traverse from source device 102 to destination device 104).

[0053] In another example, link 134 may include a storage medium configured to store the encoded content generated by source device 102. In this example, destination device 104 may be configured to access the storage medium. The storage medium may include various locally accessible data storage media such as Blu-ray discs, DVDs, CD-ROMs, flash memory, or other suitable digital storage media for storing encoded content.

[0054] In another example, link 134 may include a server or another intermediate storage device configured to store the encoded content generated by source device 102. In this example, destination device 104 may be configured to access the encoded content stored on the server or other intermediate storage device. The server may be a type of server capable of storing the encoded content and transmitting the encoded content to destination device 104.

[0055] The devices described herein can be configured to communicate with each other, such as source device 102 and destination device 104. Communication can include the sending and / or receiving of information. Information can be carried in one or more messages. As an example, a first device communicating with a second device can be described as being communicatively coupled to the second device or otherwise communicatively coupled with the second device. For example, a client device and a server can be communicatively coupled. As another example, a server can be communicatively coupled to multiple client devices. As another example, any device described herein configured to perform one or more techniques of this disclosure can be communicatively coupled to one or more other devices configured to perform one or more techniques of this disclosure. In some examples, when communicatively coupled, the two devices can actively send or receive information or can be configured to send or receive information. If not communicatively coupled, any two devices can be configured to be communicatively coupled to each other, such as according to one or more communication protocols that comply with one or more communication standards. Reference to "any two devices" does not mean that only two devices can be configured to be communicatively coupled to each other; rather, any two devices can include more than two devices. For example, a first device can be communicatively coupled to a second device, and the first device can be communicatively coupled to a third device. In such an example, the first device may be a server.

[0056] refer to Figure 1, source device 102 can be described as being communicatively coupled to destination device 104. In some examples, the term "communicatively coupled" can refer to a direct or indirect communication connection. In some examples, link 134 can represent a communicative coupling between source device 102 and destination device 104. A communication connection can be wired and / or wireless. A wired connection can refer to a conductive path, track, or physical medium (excluding wireless physical media) through which information can be transmitted. A conductive path can refer to any conductor of any length (such as a conductive pad, a conductive via, a conductive plane, a conductive track, or any conductive medium). A direct communication connection can refer to a connection in which an intermediate component resides between two communicatively coupled components. An indirect communication connection can refer to a connection in which at least one intermediate component resides between two communicatively coupled components. Two communicatively coupled devices can communicate with each other over one or more different types of networks (e.g., wireless networks and / or wired networks) according to one or more communication protocols. In some examples, two communicatively coupled devices can associate with each other through an association process. In other examples, two communicatively coupled devices can communicate with each other without involving an association process. For example, a device such as source device 102 can be configured to unicast, broadcast, multicast, or otherwise send information (e.g., encoded content) to one or more other devices (e.g., one or more destination devices, including destination device 104). The destination device 104 in this example can be described as being communicatively coupled to each of the one or more other devices. In some examples, a communication connection can enable the sending and / or receiving of information. For example, according to the technology of the present disclosure, a first device communicatively coupled to a second device can be configured to send information to the second device and / or receive information from the second device. Similarly, according to the technology of the present disclosure, the second device in this example can be configured to send information to the first device and / or receive information from the first device. In some examples, the term "communicatively coupled" can refer to a temporary, intermittent, or permanent communication connection.

[0057] Any device described herein (such as source device 102 and destination device 104) can be configured to operate according to one or more communication protocols. For example, source device 102 can be configured to communicate with destination device 104 (e.g., receive information from destination device 104 and / or send information to destination device 104) using one or more communication protocols. In such an example, source device 102 can be described as communicating with destination device 104 via a connection. The connection can be compliant with or otherwise conform to a communication protocol. Similarly, destination device 104 can be configured to communicate with source device 102 (e.g., receive information from source device 102 or send information to source device 102) using one or more communication protocols. In such an example, destination device 104 can be described as communicating with source device 102 via a connection. The connection can be compliant with or otherwise conform to a communication protocol.

[0058] As used herein, the term "communication protocol" may refer to any communication protocol, such as a communication protocol that complies with a communication standard. As used herein, the term "communication standard" may include any communication standard, such as a wireless communication standard and / or a wired communication standard. A wireless communication standard may correspond to a wireless network. As an example, a communication standard may include any wireless communication standard corresponding to a wireless personal area network (WPAN) standard (such as Bluetooth (e.g., IEEE 802.15), Bluetooth low energy (BLE) (e.g., IEEE 802.15.4)). As another example, a communication standard may include any wireless communication standard corresponding to a wireless local area network (WLAN) standard (such as WI-FI (e.g., any 802.11 standard, such as 802 / 11a, 802.11b, 802.11c, 802.11n, or 802.11ax)). As another example, a communication standard may include any wireless communication standard corresponding to a wireless wide area network (WWAN) standard (such as 3G, 4G, 4G LTE, or 5G).

[0059] refer to Figure 1, the content encoder 108 can be configured to encode the graphics content. In some examples, the content encoder 108 can be configured to encode the graphics content into one or more video frames. When the content encoder 108 encodes the content, the content encoder 108 can generate a bitstream. The bitstream can have a bit rate such as bits / time unit, where the time unit can be any time unit such as seconds or minutes. The bitstream can include a sequence of bits that can form a coded representation of the graphics content and associated data. To generate the bitstream, the content encoder 108 can be configured to perform an encoding operation on pixel data (such as pixel data corresponding to a shaded texture atlas). For example, when the content encoder 108 performs an encoding operation on image data (e.g., one or more blocks of a shaded texture atlas) provided as input to the content encoder 108, the content encoder 108 can generate a series of coded images and associated data. The associated data can include a set of codec parameters (such as a quantization parameter (QP)).

[0060] Motion estimation is the process of analyzing multiple two-dimensional (2D) images and generating motion vectors that describe the movement of regions from one image to another. Essentially, motion estimation generates motion vectors that can describe how objects move within certain parts of an image. Motion vectors have various uses, including video compression, post-processing effects such as motion blur, and frame extrapolation or interpolation. To reduce the rendering workload placed on the GPU, virtual reality (VR) or augmented reality (AR) systems can utilize motion estimation to extrapolate frames from previously rendered content. By doing so, it can allow the GPU to render frames at a reduced rate, displaying the extrapolated frames to the user instead of the rendered content. Motion estimation can be useful in VR or AR systems because there is a strong drive to reduce the rendering workload (e.g., on the GPU). The present disclosure can reduce the rendering workload by rendering fewer frames on the GPU and using motion estimation to fill in the gaps in the images or motion vectors. Furthermore, while motion estimation can be performed on video content, the motion estimation described herein can utilize rendered content being rendered in real time on the GPU.

[0061] In some cases, motion estimation has difficulty handling repeating patterns in the input image. For example, rendered content often has repeating patterns more frequently than other content (e.g., photographic content) because rendered content uses texture mapping with repeating patterns. In some respects, motion estimation techniques may have problems with repeating patterns because these techniques attempt to match objects that move from one frame to another. Because rendered content may, in some cases, precisely repeat patterns, false motion matches may occur when motion estimation incorrectly attempts to match motion. For example, due to repeating patterns, motion estimation may skip motion cycles and result in incorrect mapping for the next element, generating incorrect motion estimates. Furthermore, because many areas of an image may match equally well, some systems may have difficulty correctly identifying motion. In these areas of incorrect motion estimation, erroneous motion vectors may be generated, which can cause significant corruption in use cases that rely on accurate motion identification. Because rendered content may contain liberal use of repeating textures and patterns, incorrect motion estimation may become an increasingly significant problem as the use cases for rendered content motion estimation increase.

[0062] Some aspects of the present disclosure may provide methods for identifying erroneous motion vectors. By correctly identifying erroneous motion vectors, they can be removed to produce a more accurate motion estimate. In some aspects, motion estimation according to the present disclosure may be in addition to or in addition to the general motion estimation process. Therefore, some aspects of the present disclosure may perform the motion estimation process multiple times (e.g., twice). In these cases, motion estimation may be performed with input perturbations, which may result in an overall improvement in motion estimation. This motion estimation with input perturbations may be performed twice, with one pass being performed on the original input image and a second pass being performed on a perturbed version of the input image. In some aspects, the first pass may be an unperturbed or strict pass by sending the input image and generating resultant motion vectors. The second pass may be a perturbed pass, in which the motion vectors of the input image are perturbed in some manner. When both passes are completed, the resultant motion vectors may be compared to determine true motion and invalid motion (e.g., caused by a repeating pattern). In some cases, the first and second passes may be performed simultaneously or in parallel. Resultant vectors that match between the original and perturbed passes are identified as valid, and vectors that do not match are discarded as invalid.

[0063] The present disclosure can perform input perturbations in a variety of ways. In some aspects, the present disclosure can introduce enough new irregularities into the image so that it can disrupt motion estimation. For example, by introducing new irregularities into the motion vectors, the motion estimation can affect the motion vectors in certain areas of the input image (e.g., areas with a repeating pattern). In some cases, this irregularity can be an incremental or differential value of the motion vectors. In these cases, there should not be so much incremental or differential value that the motion tracking of the real object is lost. Therefore, the present disclosure can find a balance between adding enough noise to perturb the input image but not too much to affect the actual motion. As described above, the present disclosure can compare two passes, such as an unperturbed pass and a perturbed pass. Areas where the motion estimates do not line up between the two passes are identified as irrelevant or erroneous motion vectors. In some examples, there may be too much perturbation that the motion estimate is significantly changed to the point of affecting the actual motion. In fact, too much perturbation may cause the motion in some areas of the input image to not accurately follow the actual motion, but rather some artifacts of the erroneous repeating pattern.

[0064] Figure 2 An example of motion estimation 200 according to the present disclosure is shown. Figure 2 It is shown that both the rendered content block 202 and the RGB perturbed texture block 204 can be input to the color space conversion passes 206 and 208. Figure 2 As shown, color space conversions 206 and 208 may lead to motion estimation 210 and 212 , which may lead to delta or differential analysis 214 , which may then lead to motion vector calculation 216 . Figure 2 The color space conversion 206 / 208 in the above example can take into account the aforementioned perturbations when converting to luminance (Y), first chrominance (U), and second chrominance (V) (YUV) values. The conversion from RGB to YUV can be an efficient way to perform perturbations on the GPU. Figure 2 As shown, some aspects may first sample the rendered content, then perturb the RGB values, and then perform a YUV conversion based on these values. Some aspects of the present disclosure may not be related to the type of YUV standard used. For example, the type of YUV conversion may depend on the YUV standard followed. In other aspects, this conversion can be performed in addition to the color space conversion by calculating noise on a pixel-by-pixel basis. In addition, an efficient way to perform the perturbation may be to input the noise as another texture.

[0065] In some aspects, perturbation can allow the GPU to determine motion vectors and then render subsequent frames. Effective perturbation can adjust the input enough to destroy false area or feature matches caused by repeating patterns, but may not affect the extent to which the recognition of the input to real motion is weakened. In some aspects of the present disclosure, the motion estimation path of a VR or AR framework for performing frame extrapolation can be implemented on certain platforms (e.g., the SDM845 platform). As described above, the method for performing perturbation can be color space conversion. The rendered content can typically be in RGB color space and the motion estimation can be in YUV color space. Therefore, before the present disclosure performs motion estimation, a color space conversion can be performed. For example, a color space conversion can be performed on the GPU during the first pass or an unperturbed pass. The present disclosure can also sample a texture containing values ​​of how the input will be perturbed. Therefore, the present disclosure can perform this adjustment during the color conversion and then continue to perform motion estimation.

[0066] In some cases, the aforementioned perturbation can be accomplished by passing an additional noise texture to the existing RGB to YUV conversion (e.g., by using a conversion shader that applies uniformly distributed random RGB values ​​of a certain amplitude). In some aspects, the amplitude can be added to each color channel of the input image before the color space conversion. In some aspects, within the color space conversion, additive noise can be performed and then the color space conversion can be performed on the adjusted values. The conversion shader can apply multiple different amplitudes, such as a 5% amplitude. The amplitude value can be predetermined or arrived at experimentally by determining when the noise can eliminate too many reasonable vectors. In practice, this amplitude value can be flexible, as the same amplitude value may not be well suited for every image.

[0067] In some aspects, rather than simply applying noise of a constant amplitude, the amplitude can be varied based on the variance of the input image. For example, if there is an input image with high contrast areas in a particular region, the present disclosure can apply a higher degree of noise texture. In other instances, if there is an image with low variance and softer features, the present disclosure can apply a lower level of noise in the perturbation pass. When the amplitude of the noise is varied based on the variance of the image, some aspects of the present disclosure can obtain improved results. For example, if the present disclosure determines the variance of a region and uses it to determine the amount of noise to apply, then improved motion estimation can be obtained. Therefore, by measuring the variance of each local region of the image and using the variance to determine how much noise to apply, the present disclosure can improve the ability to detect false motion vectors and reduce the possibility of accidentally discarding real motion. In fact, based on the real input image, the present disclosure can obtain more effective results with more or less noise.

[0068] As described above, the amount of variance applied by the present disclosure can be content-dependent. If too much noise is applied, feature recognition of real features may be disrupted. In some examples, depending on the content of the image, a 10% amplitude noise may be too high. For example, by applying too much noise to a certain area, the result may be too many erroneous motion vectors in that area. As the noise begins to interfere with actual object tracking, some legitimate motion vectors may also be identified as erroneous. Therefore, if too much perturbation is applied, actual object tracking may be disrupted. For example, applying too much noise may cause some areas of the input image with legitimate motion to mismatch; for example, the delta or differential may be too high, which can lead to the erroneous conclusion that legitimate motion vectors are erroneous vectors. Simply put, if the amount of noise is too high and the adjustment is too large, the noise may deviate too much from the actual vector and the noise will drown out the real features. Therefore, the present disclosure can evaluate the input image to appropriately adjust the amount of perturbation to be applied. In fact, some aspects of the present disclosure can adjust the amount of noise or perturbation applied to the motion vectors based on the specific content of the input image.

[0069] Figure 3A and 3B Images 300 and 310 are shown, respectively, on which motion estimation is performed. Since motion estimation can be performed on images 300 or 310, these images may be referred to as motion estimation or motion estimated images. Figure 3A As shown, image 300 includes a repeating background 302 and an inset texture image 304. In some aspects, repeating background 302 can be stationary. Inset texture image 304 can move in a variety of different directions, for example, across repeating background 302. Image 300 illustrates one example of motion estimation that can be performed on a hardware platform (e.g., the SDM845 platform) according to the present disclosure.

[0070] In some aspects, image 300 can include a strict or unperturbed pass by processing the input image or embedded texture image 304 and obtaining the resulting motion vectors. In these aspects, the "X" in the repeating background 302 may appear to be moving to the right or left because it will match other "X"s to the right or left. In other aspects, image 300 can include a perturbed pass, in which the motion vectors of the input image or embedded texture image 304 are perturbed in some manner, resulting in a perturbed image. In these aspects, the input image or embedded texture image 304 can be shifted slightly, for example, to the right or left, to produce the perturbed image, making it less likely that the same "X" will match in the repeating background 302. For example, when image 304 is perturbed to produce the perturbed image, different incorrect "X" matches may be present compared to a strict or unperturbed pass.

[0071] Figure 3B An image 310 is shown which is the result of the perturbation pass. For example, the image 310 may be a Figure 3A The image 300 in is the result of applying the perturbation. Figure 3B , the disturbance in image 310 can be shown by the repeating background 312 shown as gray and the embedded texture image 314. In contrast, Figure 3A 3. The perturbation pass is shown with the repeating background 302 and the embedded texture image 304 displayed as black before the perturbation pass. In practice, the entire image 300 may be perturbed during the perturbation pass, which may cause the entire image 310 to be perturbed (e.g., displayed as gray instead of black). The results of the perturbation on the image may manifest in various ways (e.g., the image may change or fade in color). For example, the image 310 may fade in color compared to the image 300. In some aspects, the results of the perturbation may be applied to the entire image. In other aspects, the results of the perturbation may be applied to a specific portion of the image.

[0072] In some aspects, when the "X" value in the image is disturbed or shaken by the disturbance, the background "X" is more likely to move in a different direction and not match the previous pass. By doing so, the present disclosure can compare the delta or difference between the strict pass and the perturbation pass. In essence, the present disclosure can compare the common motion vectors between the strict pass and the perturbation pass. These common motion vectors may correspond to real object motion. The present disclosure can also perform more than two passes, so that in addition to the strict pass and the perturbation pass, there can be multiple passes. In addition, when performing more than two passes, multiple passes can be performed in parallel. Although performing more than two passes can provide a better estimate, the indirect cost may limit the number of passes that can be performed.

[0073] As described above, some examples of the present disclosure may not render every frame. For example, a GPU according to the present disclosure may render every other frame and still utilize motion vectors for the rendered content. In some aspects, due to the demands of VR rendering, the GPU may be resource-constrained, so not rendering every frame can save GPU resources and achieve a specific frame rate by offloading rendering work from the GPU. As previously described, the present disclosure may use motion vectors and resulting motion estimates as a substitute or replacement for rendering every frame. By doing so, the present disclosure can save power and performance at the GPU. In addition, this can allow the present disclosure to render higher quality frames at the GPU.

[0074] In some aspects of the present disclosure, when motion estimation is performed, the result may be a single vector solution. In these cases, a threshold can be set for the amount of deviation between the unperturbed or strict pass and the perturbed pass to determine whether a vector is reasonable or incorrect. As described above, this can be referred to as calculating a delta or differential, and the delta or differential value can be set to any preferred value. In one example of using deltas to calculate motion estimates, if the difference between the passes for a certain vector is less than the delta, it can be assumed that the vector is a motion vector from the unperturbed pass. In another example, if the difference between the passes for a certain vector is greater than the delta, it can be assumed that there is zero motion. Aspects of the present disclosure can also compare adjacent vectors and determine their motion. Some aspects of the present disclosure can also assume that it is better to have no motion estimate than to have an incorrect estimate. In essence, an assumption can be made that no motion estimate is better than an incorrect estimate.

[0075] The present disclosure can provide a variety of different methods for applying perturbations to motion estimation. For example, as described above, the variance can include local image variances calculated based on input frame data. In some aspects, the variance can be generated as additional data during motion estimation. Thus, the variance data for the perturbed motion estimation pass can be taken from the unperturbed motion estimation pass. Furthermore, the variance data for the perturbed motion estimation pass can be approximated by utilizing variance data from a previous frame. In some aspects, this can be done to avoid losing parallelism on the motion estimation pass for the current frame.

[0076] Some aspects of the present disclosure may provide shaders that apply the aforementioned noise perturbations. For example, the present disclosure may use the following code for these types of shaders:

[0077]

[0078] In the example code above, texArray may be the input image, noiseTexture may be the noise perturbation texture, perturbationFactor may be the scaling factor for the amount of perturbation applied, vec3 may be the input image texture coordinates, and fragColor may be the output fragment color.

[0079] The present disclosure may also provide shaders that apply noise perturbations that take into account local variance. For example, the present disclosure may use the following code for these types of shaders:

[0080]

[0081]

[0082] In the example code above, texArray can be the input image, noiseTexture can be the noise perturbation texture, varianceTexture can be the variance data texture, varianceSampleRange can be the size of the region of the variance texture to sample, varianceSampleSpread can be the distance between the variance texture sample point locations, perturbationFactor can be the scaling factor for the amount of perturbation to be applied, vec3 can be the input image texture coordinates, and fragColor can be the output fragment color.

[0083] The present disclosure may also process two arrays of motion vectors to detect deviations. For example, the resulting motion vectors from the two aforementioned passes may be compared and utilized. In some aspects, the threshold may be based on a tuning parameter for a particular use case set based on the use case's deviation from incorrect motion estimates. As described above, in a use case utilizing prototype VR, it is better to not generate a motion estimate than to generate an incorrect motion estimate. Therefore, some aspects of the present disclosure may set the threshold to a very small value (e.g., 0.001). The present disclosure may use the following code to process two arrays of motion vectors to detect deviations:

[0084]

[0085]

[0086] Figure 4A Another example of motion estimation 400 according to the present disclosure is shown. Figure 4A As shown, motion estimation 400 includes a frame 402, a repeating background 404, an embedded texture image 410, at least one first motion vector 412, at least one second motion vector 414, and a difference 416 between the at least one first motion vector 412 and the at least one second motion vector 414. Figure 4A As shown, in some aspects, at least one first motion vector 412 can be generated in a first subset of frames 402. Figure 4AAs shown, in some aspects, the first frame subset can be located above an embedded texture image 410 in a portion of a repeating background 404. In other aspects, the first frame subset can be located in another portion of frame 402. At least one first motion vector 412 can provide a motion estimate for the image data in the first frame subset. In some aspects, the image data in the first frame subset can be perturbed. Furthermore, at least one second motion vector 414 can be generated based on the perturbed image data in the first frame subset. The at least one second motion vector 414 can provide a second motion estimate for the image data in the first frame subset. Furthermore, the at least one first motion vector 412 can be compared to the at least one second motion vector 414. In some aspects, comparing the at least one first motion vector 412 to the at least one second motion vector 414 can include determining a difference 416 between the at least one first motion vector 412 and the at least one second motion vector 414. The difference 416 can be determined to be less than or greater than a threshold. In some aspects, the above comparison can be expressed as a function, such as the formula: f(v1, v2) = |v1–v2| <threshold。

[0087] In some aspects, at least one third motion vector for motion estimation of image data in the first subset of frames can be determined based on a comparison between at least one first motion vector 412 and at least one second motion vector 414. Furthermore, when a difference 416 is less than a threshold, the at least one third motion vector can be set to the at least one first motion vector. Difference 416 can also be referred to as a delta analysis. As described herein, in some aspects, difference 416 can be an absolute value. Based on the above, the at least one third motion vector can be expressed as follows: when |v1–v2| < threshold, v3 = v1. In some aspects, when difference 416 is greater than or equal to the threshold, the at least one third motion vector can be set to have a zero motion value. Based on this, the at least one third motion vector can also be expressed as follows: when |v1–v2| ≥ threshold, v3 = 0.

[0088] In some cases, a second difference between the at least one third motion vector and one or more adjacent vectors surrounding or near the at least one third motion vector may be determined. Figure 4A As shown, one or more neighboring vectors surrounding or near the at least one third motion vector may be used to determine a second difference between the third motion vector and the neighboring vector. Furthermore, if the difference is greater than a threshold and / or the second difference is less than a threshold, the at least one third motion vector may be set based on the one or more neighboring vectors. Furthermore, the aforementioned threshold may be based on a motion estimation deviation of the image data in the first frame subset.

[0089] Figure 4B Another example of motion estimation 450 according to the present disclosure is shown. Figure 4BAs shown in , the motion estimate 450 includes a frame 452, a repeating background 454, an embedded texture image 460, at least one first motion vector 462, at least one second motion vector 464, a difference between the at least one first motion vector 462 and the at least one second motion vector 464, and at least one fourth motion vector 472. Figure 4B In, such as combined Figure 4A As described in , at least one first motion vector 462 in a first subset of frames 452 may be generated, wherein the at least one first motion vector 462 may provide a first motion estimate for image data in the first subset of frames. Figure 4B As shown, in some aspects, the first frame subset can be located above an embedded texture image 460 in a portion of a repeating background 454. In other aspects, the first frame subset can be located in another portion of frame 452. In some cases, the image data in the first frame subset can be perturbed. Furthermore, at least one second motion vector 464 can be generated based on the perturbed image data in the first frame subset, where the at least one second motion vector 464 can provide a second motion estimate for the image data in the first frame subset. Furthermore, the at least one first motion vector 462 can be compared to the at least one second motion vector 464. In some aspects, comparing the at least one first motion vector 462 to the at least one second motion vector 464 can include determining a difference 466 between the at least one first motion vector 462 and the at least one second motion vector 464. The difference 466 can be determined to be less than or greater than a threshold. In some aspects, based on the comparison between the at least one first motion vector 462 and the at least one second motion vector 464, at least one third motion vector can be determined for the motion estimate of the image data in the first frame subset.

[0090] like Figure 4B As shown in , at least one fourth motion vector 472 may be determined for motion estimation of a second subset of frames 452. Figure 4B As shown in , in some aspects, the second subset of frames can be located below the first subset of frames in the portion of the repeating background 454 and above the embedded texture image 460. In some aspects, when the difference 466 is greater than or equal to the threshold, at least one fourth motion vector 472 can be determined. Figure 4B As shown in , in some aspects, the second subset of frames can be different from the first subset of frames. Additionally, at least one third motion vector can be determined based on at least one first motion vector 462, at least one second motion vector 464, and at least one fourth motion vector 472.

[0091] In some aspects, perturbing the image data in the first subset of frames may include adjusting the red (R), green (G), and blue (B) (RGB) values ​​of the image data by a value (e.g., m). In some cases, m may not be equal to zero. Furthermore, m may be less than or equal to 5% and greater than or equal to -5%. Furthermore, the perturbation amount may perturb the image data in the first subset of frames, and the perturbation amount may be adjusted based on the local variance of the RGB values ​​of the image data. Furthermore, the local variance of the RGB values ​​of the image data may be approximated based on the previous variance of the RGB values. In other aspects, the perturbation amount may perturb the image data in the first subset of frames, and the perturbation amount may be adjusted based on the local variance of the luminance (Y), first chrominance (U), and second chrominance (V) (YUV) values ​​of the image data. Furthermore, the image data from the RGB image data may be converted to YUV image data, wherein the image data may be perturbed before converting the RGB image data to the YUV image data.

[0092] Figure 5A and Figure 5B Examples of motion estimation 500 and 550 according to the present disclosure are shown. Figure 5A and Figure 5B As shown in FIG, motion estimation 500 and motion estimation 550 include corresponding motion vectors to assist motion estimation. More specifically, Figure 5A A motion estimate 500 without a perturbation pass is shown. Thus, the motion estimate 500 may include regions 502 and 504 of incorrect motion. Figure 5B A motion estimate 550 including a perturbation pass is shown. Likewise, the motion estimate 550 may not include regions of erroneous motion because these motion vectors have been identified as incorrect and removed.

[0093] Figure 6 An example flowchart 600 of an example method according to one or more techniques of this disclosure is shown. The method can be performed by a device or a GPU. The method can assist the GPU in the process of motion estimation.

[0094] At 602, as combined Figure 2 Figure 3 Figure 4A and Figure 5B As described in the example of , the GPU may generate at least one first motion vector in a first subset of frames. For example, the first motion vector may provide a first motion estimate for image data in the first subset of frames. At 604, as in conjunction with Figure 2 Figure 3 Figure 4A and Figure 5BAs described in the example of , the GPU may perturb the image data. In some aspects, the GPU may perturb all image data in the frame. In other aspects, the GPU may perturb image data in a first subset of the frame. In other aspects, the GPU may perturb image data outside the first subset of the frame. Additionally, at 606, as in conjunction with Figure 2 Figure 3 Figure 4A and Figure 5B As described in the example of , the GPU can generate at least one second motion vector based on the perturbed image data. In some aspects, the second motion vector can provide a second motion estimate for the image data. At 608, as in conjunction with Figure 2 Figure 3 Figure 4A and Figure 5B As described in the example of , the GPU can compare the first motion vector with the second motion vector. In addition, at 610, as in conjunction with Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example, the GPU may determine at least one third motion vector for motion estimation of the image data based on a comparison between the first motion vector and the second motion vector.

[0095] In some aspects, such as combining Figure 2 As described in the example of FIG, when comparing a first motion vector to a second motion vector, the GPU can determine the difference between the first motion vector and the second motion vector. The difference can also be referred to as a delta analysis. In some cases, the difference can be an absolute value. In addition, as in conjunction with Figure 2 As further described in the example of , the GPU can determine whether the difference is less than a threshold. Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of , when the difference is less than the threshold, the third motion vector can be set to the first motion vector. Figure 2 As described in the example of , when the difference is greater than the threshold, the GPU may also determine at least one fourth motion vector for motion estimation of the second subset of frames. In addition, the second subset of frames may be different from the first subset of frames. In some aspects, such as in conjunction with Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example in, the third motion vector can be determined based on the determined first motion vector, second motion vector and fourth motion vector.

[0096] In addition, if combined Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5BAs described in the example of , in some aspects, when the difference is greater than the threshold, the third motion vector can be set to have a zero motion value. In some aspects, as in conjunction with Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of , the GPU can determine the third motion vector and one or more neighboring vectors around the third motion vector. In these aspects, when the difference is greater than a threshold and the second difference is less than a threshold, the third motion vector can be set based on the one or more neighboring vectors.

[0097] In other aspects, such as combining Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of , the aforementioned threshold value can be based on the motion estimation tolerance of the image data in the first subset of frames. Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of FIG. 1 , perturbing the image data in the first subset of frames may include adjusting the amplitudes of the RGB values ​​of the image data by m. In some cases, m may not be equal to zero. Furthermore, the absolute value of m may be between -5% and 5%, such that m may be less than or equal to 5% and greater than or equal to -5%.

[0098] In other aspects, such as combining Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of , the image data in the first subset of the frame can be perturbed by a perturbation amount, wherein the perturbation amount can be adjusted based on the local variance of the RGB values ​​of the image data. Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of , the local variance of the RGB values ​​of the image data can be based on the first motion vector for the image data in the first subset of the frame. Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of , the local variance of the RGB values ​​of the image data can be approximated based on the previous variance of the RGB values. Figure 2 Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example of , the image data in the first subset of the frame can be perturbed by a perturbation amount, wherein the perturbation amount can be adjusted based on the local variance of the YUV values ​​of the image data. In other aspects, such as in combination Figure 2Figure 3 Figure 4A 、 Figure 4B and Figure 5B As described in the example in , the GPU may convert the image data from RGB image data to YUV image data, wherein the image data is perturbed before converting the RGB image data to YUV image data.

[0099] In one configuration, an apparatus for motion estimation is provided. The apparatus may be a motion estimation device in a GPU. In one aspect, the motion estimation device may be processing unit 120 within device 104 or may be some other hardware within device 104 or another device. The apparatus may include means for generating at least one first motion vector in a first subset of frames. The first motion vector may provide a first motion estimate for image data in the first subset of frames. The apparatus may also include means for perturbing the image data in the first subset of frames. Furthermore, the apparatus may also include means for generating at least one second motion vector based on the perturbed image data in the first subset of frames. In some aspects, the second motion vector may provide a second motion estimate for the image data in the first subset of frames. The apparatus may also include means for comparing the first motion vector with the second motion vector. Furthermore, the apparatus may include means for determining at least one third motion vector for use in the motion estimation of the image data in the first subset of frames based on the comparison between the first motion vector and the second motion vector.

[0100] In some aspects, the component for comparing the first motion vector to the second motion vector can be configured to determine a difference between the first motion vector and the second motion vector, and determine whether the difference is less than a threshold. The device may also include a component for determining at least one fourth motion vector for motion estimation for a second subset of frames when the difference is greater than the threshold. The second subset of frames may be different from the first subset of frames, wherein the third motion vector is determined based on the first motion vector, the second motion vector, and the fourth motion vector. In addition, the device may also include a component for determining a second difference between the third motion vector and one or more neighboring vectors surrounding the third motion vector. When the difference is greater than the threshold and the second difference is less than the threshold, the third motion vector may be set based on the one or more neighboring vectors. In addition, the component for perturbing the image data in the first subset of frames may be configured to adjust the amplitude of the RGB values ​​of the image data by m, where m is not equal to zero. In addition, the device may include a component for converting the image data from RGB image data to YUV image data, wherein the image data is perturbed before converting the RGB image data to YUV image data.

[0101] The subject matter described herein can be implemented to achieve one or more potential benefits or advantages. For example, a GPU can use the described techniques to reduce the amount of rendering work. The system described herein can utilize motion estimation to extrapolate frames from previously rendered content. By doing so, this can allow the GPU to render frames at a reduced rate, displaying the extrapolated frames instead of the rendered content. Thus, the present disclosure can reduce the rendering workload by rendering fewer frames on the GPU and using motion estimation to fill in gaps in images or motion vectors to reduce the rendering workload. As a result, the present disclosure can save power and performance at the GPU, and can also render higher quality frames at the GPU. In addition, the present disclosure can also reduce the cost of rendering content.

[0102] According to the present disclosure, the term "or" can be interpreted as "and / or" unless the context requires otherwise. In addition, although phrases such as "one or more" or "at least one" have been used with some features of the present disclosure and not with others, features that do not use such language can be interpreted as having such an implied meaning unless the context requires otherwise.

[0103] In one or more examples, the functionality described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" has been used throughout this disclosure, such a processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any functionality, processing unit, technique, or other module described herein is implemented in software, the functionality, processing unit technique, or other module described herein may be stored on or transmitted through a computer-readable medium as one or more instructions or code. Computer-readable media may include computer data storage media or communication media, including media that facilitate the transfer of computer programs from one place to another. In this manner, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures to implement the techniques described in this disclosure. By way of example, but not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above are also intended to be included within the scope of computer-readable media. A computer program product may include a computer-readable medium.

[0104] The code may be executed by one or more processors (e.g., one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), arithmetic logic units (ALUs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits). Thus, the term "processor," as used herein, may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0105] The techniques of this disclosure can be implemented in various devices or apparatuses, including wireless handsets, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware. Instead, as described above, the various units can be combined in any hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above in combination with appropriate software and / or hardware.

[0106] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for motion estimation in a graphics processing unit (GPU), comprising: generating at least one first motion vector in a first subset of frames, the at least one first motion vector providing a first motion estimate for image data in the first subset of frames; to perturb image data in a first subset of the frames; generating at least one second motion vector based on the perturbed image data in the first subset of frames, the at least one second motion vector providing a second motion estimate for the image data in the first subset of frames; comparing the at least one first motion vector with the at least one second motion vector by determining whether a difference between the at least one first motion vector and the at least one second motion vector is less than a threshold; determining at least one fourth motion vector for motion estimation of a second subset of frames when the difference is greater than the threshold, the second subset of frames being different from the first subset of frames; as well as At least one third motion vector for motion estimation of image data in the first subset of the frame is determined based on a comparison between the at least one first motion vector and the at least one second motion vector, wherein when the difference is less than the threshold, the at least one third motion vector is set to the at least one first motion vector, wherein when the absolute value of the difference is greater than or equal to the threshold, the at least one third motion vector is set to have a motion value of zero, wherein the at least one third motion vector is determined based on the at least one first motion vector, the at least one second motion vector, and the at least one fourth motion vector.

2. The method according to claim 1, further comprising: determining a second difference between the at least one third motion vector and one or more neighboring vectors surrounding the at least one third motion vector; Wherein when the difference is greater than the threshold and the second difference is less than the threshold, the at least one third motion vector is set based on the one or more neighboring vectors. The method of claim 1 , wherein the threshold is based on a motion estimation tolerance of image data in the first subset of the frames. 4 . The method of claim 1 , wherein perturbing the image data in the first subset of frames comprises adjusting the amplitudes of the RGB values ​​of red (R), green (G), and blue (B) of the image data by m, wherein m is not equal to 0. The method according to claim 4 , wherein m is less than or equal to 5% and greater than or equal to −5%. 6 . The method of claim 1 , wherein the image data in the first subset of the frames is perturbed by a perturbation amount, and the perturbation amount is adjusted based on a local variance of red (R), green (G), and blue (B) RGB values ​​of the image data. 7 . The method of claim 6 , wherein the local variance of the RGB values ​​of the image data is based on a first motion estimate of the image data in the first subset of the frames. The method of claim 6 , wherein the local variance of the RGB values ​​of the image data is approximated based on a previous variance of the RGB values.

9. The method of claim 1 , wherein the image data in the first subset of the frames is perturbed by a perturbation amount, and the perturbation amount is adjusted based on a local variance of YUV values ​​of luminance (Y), first chrominance (U), and second chrominance (V) of the image data.

10. The method of claim 1, further comprising converting the image data from red (R), green (G), and blue (B) image data into YUV image data of luma (Y), first chroma (U), and second chroma (V), wherein the image data is perturbed before converting the RGB image data into the YUV image data.

11. A device for motion estimation in a graphics processing unit (GPU), comprising: Memory; as well as at least one processor coupled to the memory and configured to: generating at least one first motion vector in a first subset of frames, the at least one first motion vector providing a first motion estimate for image data in the first subset of frames; to perturb image data in a first subset of the frames; generating at least one second motion vector based on the perturbed image data in the first subset of frames, the at least one second motion vector providing a second motion estimate for the image data in the first subset of frames; comparing the at least one first motion vector with the at least one second motion vector by determining whether a difference between the at least one first motion vector and the at least one second motion vector is less than a threshold; determining at least one fourth motion vector for motion estimation of a second subset of frames when the difference is greater than the threshold, the second subset of frames being different from the first subset of frames; as well as At least one third motion vector for motion estimation of image data in the first subset of the frame is determined based on a comparison between the at least one first motion vector and the at least one second motion vector, wherein when the difference is less than the threshold, the at least one third motion vector is set to the at least one first motion vector, wherein when the absolute value of the difference is greater than or equal to the threshold, the at least one third motion vector is set to have a motion value of zero, wherein the at least one third motion vector is determined based on the at least one first motion vector, the at least one second motion vector, and the at least one fourth motion vector.

12. The apparatus of claim 11 , wherein the at least one processor is further configured to: determining a second difference between the at least one third motion vector and one or more neighboring vectors surrounding the at least one third motion vector; Wherein when the difference is greater than the threshold and the second difference is less than the threshold, the at least one third motion vector is set based on the one or more neighboring vectors.

13. The apparatus of claim 11, wherein the threshold is based on a motion estimation tolerance of image data in the first subset of the frames.

14. The apparatus of claim 11 , wherein the image data in the first subset of perturbed frames comprises the at least one processor being further configured to adjust amplitudes of red (R), green (G), and blue (B) RGB values ​​of the image data by m, wherein m is not equal to 0. The apparatus according to claim 14 , wherein m is less than or equal to 5% and greater than or equal to −5%. 16 . The apparatus of claim 11 , wherein the image data in the first subset of the frames is perturbed by a perturbation amount, and the perturbation amount is adjusted based on a local variance of red (R), green (G), and blue (B) RGB values ​​of the image data.

17. The apparatus of claim 16, wherein the local variance of RGB values ​​of the image data is based on a first motion estimate of image data in the first subset of the frames.

18. The apparatus of claim 16, wherein the local variance of the RGB values ​​of the image data is approximated based on a previous variance of the RGB values.

19. The apparatus of claim 11, wherein the image data in the first subset of the frames is perturbed by a perturbation amount, and the perturbation amount is adjusted based on a local variance of YUV values ​​of luminance (Y), first chrominance (U), and second chrominance (V) of the image data.

20. The apparatus of claim 11, wherein the at least one processor is further configured to convert the image data from RGB image data of red (R), green (G), and blue (B) into YUV image data of luminance (Y), a first chrominance (U), and a second chrominance (V), wherein the image data is perturbed before converting the RGB image data into the YUV image data.

21. A device for motion estimation in a graphics processing unit (GPU), comprising: means for generating at least one first motion vector in a first subset of frames, the at least one first motion vector providing a first motion estimate for image data in the first subset of frames; means for perturbing image data in a first subset of said frames; means for generating at least one second motion vector based on the perturbed image data in the first subset of frames, the at least one second motion vector providing a second motion estimate for the image data in the first subset of frames; means for comparing the at least one first motion vector to the at least one second motion vector by determining whether a difference between the at least one first motion vector and the at least one second motion vector is less than a threshold; means for determining at least one fourth motion vector for motion estimation of a second subset of frames, the second subset of frames being different from the first subset of frames, when the difference is greater than the threshold; as well as means for determining at least one third motion vector for motion estimation of image data in the first subset of the frames based on a comparison between the at least one first motion vector and the at least one second motion vector, wherein when the difference is less than the threshold value, the at least one third motion vector is set to the at least one first motion vector, wherein when the absolute value of the difference is greater than or equal to the threshold value, the at least one third motion vector is set to have a motion value of zero, wherein the at least one third motion vector is determined based on the at least one first motion vector, the at least one second motion vector and the at least one fourth motion vector.

22. A computer-readable medium storing computer-executable code for motion estimation in a graphics processing unit (GPU), comprising code for: generating at least one first motion vector in a first subset of frames, the at least one first motion vector providing a first motion estimate for image data in the first subset of frames; to perturb image data in a first subset of the frames; generating at least one second motion vector based on the perturbed image data in the first subset of frames, the at least one second motion vector providing a second motion estimate for the image data in the first subset of frames; comparing the at least one first motion vector with the at least one second motion vector by determining whether a difference between the at least one first motion vector and the at least one second motion vector is less than a threshold; determining at least one fourth motion vector for motion estimation of a second subset of frames when the difference is greater than the threshold, the second subset of frames being different from the first subset of frames; as well as At least one third motion vector for motion estimation of image data in the first subset of the frame is determined based on a comparison between the at least one first motion vector and the at least one second motion vector, wherein when the difference is less than the threshold, the at least one third motion vector is set to the at least one first motion vector, wherein when the absolute value of the difference is greater than or equal to the threshold, the at least one third motion vector is set to have a motion value of zero, wherein the at least one third motion vector is determined based on the at least one first motion vector, the at least one second motion vector, and the at least one fourth motion vector.

23. A computer program product comprising computer executable code for motion estimation in a graphics processing unit (GPU), comprising code for: generating at least one first motion vector in a first subset of frames, the at least one first motion vector providing a first motion estimate for image data in the first subset of frames; to perturb image data in a first subset of the frames; generating at least one second motion vector based on the perturbed image data in the first subset of frames, the at least one second motion vector providing a second motion estimate for the image data in the first subset of frames; comparing the at least one first motion vector with the at least one second motion vector by determining whether a difference between the at least one first motion vector and the at least one second motion vector is less than a threshold; determining at least one fourth motion vector for motion estimation of a second subset of frames when the difference is greater than the threshold, the second subset of frames being different from the first subset of frames; as well as At least one third motion vector for motion estimation of image data in the first subset of the frame is determined based on a comparison between the at least one first motion vector and the at least one second motion vector, wherein when the difference is less than the threshold, the at least one third motion vector is set to the at least one first motion vector, wherein when the absolute value of the difference is greater than or equal to the threshold, the at least one third motion vector is set to have a motion value of zero, wherein the at least one third motion vector is determined based on the at least one first motion vector, the at least one second motion vector, and the at least one fourth motion vector.

Images