Optical flow techniques and systems for accurate identification and tracking of moving objects

Abstract

Apparatuses, systems, and techniques are disclosed that can perform a pyramid optical flow process that can efficiently identify and process object boundary pixels. In the pyramid optical flow, motion vectors of pixels of image layers having a coarse resolution can be used as cues for identifying motion vectors of pixels of image layers having a higher resolution. Pixels located near a distinct boundary between foreground and background objects can receive multiple cues from lower resolution image layers to more accurately identify matching pixels on different image layers of the pyramid.

Description

Technical Field

[0001] At least one embodiment relates to computational techniques for performing and facilitating graphics applications. For example, at least one embodiment relates to operations for identifying moving objects in an image, and more specifically, to the accurate correlation of image pixels (including pixels located near boundaries between different objects) to correct objects depicted in the foreground and background of the image. Background Technology

[0002] Optical flow is used to identify and track the apparent motion of objects depicted in an image. Typically, a moving object changes its position relative to a reference frame of the image-capturing camera (or a vantage point in a composite image) and its appearance, such as pixel intensity. Optical flow characterizes this change by tracking the intensity variations and displacement (or velocity) of pixels between two or more images captured (or rendered) at close but different times. For example, a given pixel in an image can be associated with a motion vector representing the displacement of a specified pixel relative to some other reference image (e.g., an earlier image of a nearly identical scene). Identification of motion vectors allows tracking the motion of objects at intermediate times between interpolated images, and extrapolating (predicting) the motion of objects at times beyond the available time of the existing image (future or past). The quality of motion tracking and prediction depends on the accuracy with which image pixels are segmented into regions belonging to different objects. Attached Figure Description

[0003] Figure 1A This is a block diagram of an example computer system capable of efficiently and accurately processing object boundary pixels in pyramid optical flow processing, according to at least some embodiments.

[0004] Figure 1B It is an example computing device according to at least one embodiment that can realize optical flow processing by efficiently and accurately processing the pixels of the object boundary;

[0005] Figure 2 This is an example pyramid optical flow architecture for efficient and accurate processing of object boundary pixels according to at least one embodiment;

[0006] Figure 3A The illustration shows a pyramid structure of image layers for efficiently representing multiple resolution variations of an image in optical flow processing, according to at least one embodiment.

[0007] Figure 3B The use of a pyramid structure to identify the motion of an object in optical flow processing is illustrated according to at least one embodiment;

[0008] Figure 3C The method of using a matching window to determine motion vectors during optical flow processing, according to at least one embodiment, is illustrated.

[0009] Figure 4A The pixel-superpixel association during optical flow processing of object boundary pixels according to at least one embodiment is illustrated;

[0010] Figure 4B The recognition of motion vector cues during optical flow processing of object boundary pixels is illustrated according to at least one embodiment;

[0011] Figure 5 This is a flowchart of an example method for pyramid optical flow processing according to at least one embodiment, wherein object boundary pixels are identified and processed at the highest resolution level;

[0012] Figure 6 This is a flowchart of an example method for pyramid optical flow processing according to at least one embodiment, wherein object boundary pixels are identified and processed at a reduced resolution level;

[0013] Figure 7 A block diagram of an example computer system operating according to some embodiments of the present disclosure is depicted. Detailed Implementation

[0014] The optical flow (OF) of a given image can be specified as a set of motion vectors MV(x,y), which represent the displacement of a pixel with coordinates x and y to a corresponding pixel in the reference image (e.g., a pixel depicting the same object or a part of the same object). Some pixels in the given image may have zero (or near-zero, down to noise) motion vectors, such as pixels depicting parts of stationary objects like buildings, roads, sidewalks, tree trunks, etc. Other pixels may have non-zero motion vectors, such as pixels depicting parts of moving objects like vehicles, people, animals, leaves / branches, etc. However, other pixels may not have motion vectors associated with them, such as pixels depicting objects not present in the reference image, such as vehicles that entered the scene after the reference image was taken. The motion vector of a pixel can be obtained by searching the reference image for pixels that are most similar to the given pixel (or a region of pixels that are most similar to the pixel).

[0015] Typical high-resolution images often have close to or exceed one million pixels, and identifying the motion vector of each pixel independently may be impractical. To improve computational efficiency, pyramid optical flow techniques can be used. In pyramid optical flow processing, an image is represented by multiple additional images (called image layers) with progressively decreasing resolutions. For example, a 1280x720 pixel image might have a first image layer with a resolution of 640x360, a second image layer with a resolution of 320x180 pixels, a third image layer with a resolution of 160x90 pixels, and a fourth (top) image layer with the lowest resolution of 80x45 pixels. Thus, each pixel (referred to as a superpixel in this paper) in a subsequent layer of the optical flow pyramid is associated with multiple pixels (or superpixels) in the previous image layer. Pixels in each subsequent layer can be obtained by blurring, smoothing, convolving, etc., the pixels in the previous layer. The motion vector of the top image layer can be found relatively quickly and then used as a cue (potentially a motion vector) for lower layers. For example, the motion vector associated with each pixel of the top 80x45 pixel image layer can be used as a cue to determine the motion vectors of four pixels in the third 160x90 pixel image layer, and so on. At each layer, the optical flow pyramid processing uses the cue as a starting hypothesis in order to search more accurately for the motion vectors of the next (coarser) image layer below, and so on, until the motion vectors of the original image (layer zero) are identified.

[0016] This pyramid processing can be problematic near the boundaries of various objects, such as a first object in the foreground moving at a different speed than a second object in the background. A superpixel in layer j+1 can be associated with multiple (e.g., four) superpixels in layer j (or the four pixels in the lowest zeroth layer), some of which (e.g., three) may belong to the foreground, while others (e.g., one) may belong to the background (or vice versa). As a result, a superpixel in layer j+1 can be associated with a moving object in the foreground and can provide false hints for a superpixel in layer j corresponding to a stationary background. Similarly, a superpixel including a pair of superpixels belonging to the foreground and another pair belonging to the background will provide false hints for one of the pairs. Such false hints can lead to a loss of computational efficiency (by causing optical flow processing to search for matching superpixels in incorrect regions) and even incorrect optical flow mapping, which can be exacerbated with each additional image layer.

[0017] This disclosure addresses these and other technical challenges by disclosing methods and systems for efficiently processing object boundary pixels and superpixels during pyramid optical flow processing by expanding the source of received motion vector cues to include cues from neighboring superpixels. More specifically, a given boundary (super)pixel X in layer j can receive cues not only from a superpixel Y in layer j+1 directly associated with (super)pixel X, but also from neighboring superpixels Y-1, Y+1, etc. in layer j+1. Among these additional superpixels are superpixels describing those portions of foreground and background objects far from the boundary. Subsequent extended sampling of possible motion vector cues enables the identification of a broader pool of candidate regions in the reference image (or reference image layer), where the best match for (super)pixel X is likely to be found. Thus, exploring such a broader pool allows optical flow processing to capture (super)pixel matches that would otherwise be missed.

[0018] The advantages of the disclosed technique include, but are not limited to, improved computational efficiency during optical flow pyramid processing and accurate identification of pixel motion between different images, identification of moving objects in images, and segmentation of images into regions corresponding to moving objects. Accurate determination of optical flow improves the quality of images (e.g., interpolated images) that can be generated based on existing images.

[0019] System Architecture

[0020] Figure 1A This is a block diagram of an example computer system 100 capable of efficiently and accurately processing object boundary pixels in pyramid optical flow processing, according to at least some embodiments. Figure 1A As shown, the computer system 100 for processing image data 166 may include an image processing server 101, an application server 160, and client devices 140 connected via a network 150. The network 150 may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wireless network, a personal area network (PAN), or any combination thereof.

[0021] Image processing server 101 may be (or include) a desktop computer, laptop computer, smartphone, tablet computer, local server, cloud server, dedicated graphics server, collection of multiple computing devices, distributed computing system, smart TV, augmented reality device, or any other suitable computing device (or collection of computing devices) capable of performing the techniques described herein. Application server 160 and client device 140 may similarly include any of the computing devices mentioned above. Alternatively, client device 140 may be a computing device lacking substantial computing resources but capable of accessing and communicating with image processing server 101 and / or application server 160. Client device 140 may have a graphical user interface (GUI) 142 for facilitating user interaction with client device 140, application server 160, and image processing server 101. GUI 142 may be implemented on a desktop or laptop computer screen, touchscreen, smart TV screen, or any combination thereof. GUI 142 may include any pointing device (mouse, touchpad, stylus, finger, eye movement control device, etc.), keyboard, control bar, game console, etc. GUI 142 can be used, for example, with browsers, desktop applications, mobile applications, etc., to display static and moving objects, light sources, shadows, colors, menus, etc.

[0022] Client device 140 may include a graphics application client 144 and an image processing client 146. The graphics application client 144 may be a client application provided and / or supported by a graphics application 162 running on application server 160. Graphics application 162 may be any game, art, medical, scientific, engineering, etc. For example, graphics application 162 may be an image creation application, a filmmaking application, a video game application, an engineering application, an architecture application, a flight simulation application, a scene reconstruction application, or the like. Graphics application 162 may operate in conjunction with image data generator 164. Image data generator 164 may receive settings for one or more images from graphics application 162. For example, graphics application 162 may be a video application that provides game context (e.g., the player's current position relative to a building map). Image data generator 164 may generate coordinates, trajectories (e.g., speed and direction of movement), brightness, contrast, chromaticity, intensity and color of light emitted by various light sources, etc., for individual objects. In some embodiments, image data generator 164 may include any suitable camera and / or a video camera capable of generating multiple images.

[0023] The generated image data 166 may include two or more images, which may be composite images generated by image data generator 164, real images captured by a camera and / or video camera, or any combination thereof. Image data 166 may be provided (e.g., via network 150) to image processing server 101 for generating optical flow. Optical flow should be understood as any set of data, presented in any suitable form, that identifies the displacement (represented by individual pixels or groups of pixels) of an object or part of an object in a given image relative to a different image (referred to herein as a reference image). The reference image may be any image taken (or rendered) at different times (e.g., past or future times) and may depict the same or similar scene. The difference between the image and the reference image may be due to any relative motion between the camera and objects in the scene. The image may be taken by a stationary (relative to the ground) camera or by a moving camera, such as a camera mounted on a vehicle or any other moving body. Optical flow may include any additional data, such as changes in pixel and superpixel intensity over time, timestamps, identification of moving and stationary objects, and / or any additional metadata.

[0024] Image data 166 can be used as part of the scenery in a computer game, for example, a computer game developed by a user of graphics application 162 via graphics application client 144. The task of image processing server 101 is to generate various images to support the computer game. For example, image processing server 101 may be able to generate additional images involving interpolation (e.g., slow-motion interpolation) within the time interval between when the image and the reference image were captured, extrapolation to different (e.g., later or earlier) times, extension to different vantage points, different arrangements of the same object, inclusion of additional objects, modification of light sources, etc.

[0025] The images in image data 166 can be in any digital (e.g., pixel-based or vector-based) format, including but not limited to JPEG, GIF, PNG, BMP, TIFF, CIB, DIMAP, NITF, etc. In some embodiments, image data 166 can be provided in a color space using a limited number of primary colors, such as the RGB space, but it should be understood that any other color space can also be used. Image processing server 101 can use optical flow accelerator 104 to facilitate and accelerate various operations related to generating optical flow associated with two or more images. In at least one embodiment, optical flow accelerator 104 can be implemented by one or more dedicated hardware circuits, which can be further integrated on a common integrated circuit. In some embodiments, optical flow accelerator 104 can be a combination of hardware circuitry and software modules, such as modules executed by graphics processing unit (GPU) 110, central processing unit (CPU) 130, or any other suitable processing device. In some embodiments, optical flow accelerator 104 can be implemented purely in software.

[0026] Optical flow accelerator 104 can access pyramid processing component 103 to perform cue-based motion vector search across multiple image layers of the optical flow pyramid, as described in more detail below. Cues for each progressively higher resolution image layer can be derived from motion vectors obtained for previous image layers. In some embodiments, cues for the top (lowest resolution) image layer can be stored in an external cue buffer 105, which can be coupled to optical flow accelerator 104 via an external cue interface. In some embodiments, external cue buffer 105 can be filled with some default cues for the top image layer, such as zero motion vector cues. In some embodiments, external cue buffer 105 can store user-specific cues, such as game-specific cues or cues specific to a particular game scenario. In some embodiments, external cue buffer 105 can store cues from different images, such as cues from previous images, including reference images or some other images. Optical flow accelerator 104 may include one or more motion vector buffers 107 for storing intermediate and final motion vectors determined during pyramid optical flow processing. The optical flow accelerator 104 may further include a boundary pixel processing component 108 for performing the identification of object boundary pixels and superpixels and selecting multiple cues for (super)pixels located near the boundaries of the foreground object and the visible portions of the background object, as described in more detail below.

[0027] In some embodiments, the operation of the optical flow accelerator 104 can be fully automated. In some embodiments, some operations can be controlled by a user via a GUI 109. If the user is accessing an image processing server 101 located remotely on the client device 140, at least a portion of the GUI 109 can be provided to the client device 140 as a GUI 142. The image processing client 146 can facilitate interaction between the user of the client device 140 and the optical flow accelerator 104 via an optical flow application programming interface (API) 122, which allows user actions to be input via the GUI 109. The optical flow API 122 package enables bidirectional communication between the user and the optical flow accelerator 104. Some operations of the optical flow accelerator 104 that can be controlled by the image processing client 146 may include: selecting the number of image layers in the optical flow pyramid, selecting images and reference images for optical flow processing, selecting the number of intermediate images (frames) to be generated based on the determined optical flow, selecting camera (advantage) points for the new images to be generated, etc. The Optical Flow API 122 can deploy any script, code, executable file, library, etc., or any combination thereof, specifying the sequence and parameters of operations that can be performed by the Optical Flow Accelerator 104. Although Figure 1A The image shows that the optical flow API 122 is located on the image processing server 101, but in some embodiments, the optical flow API package can be downloaded and installed on the client device 140 to facilitate user interaction with the remote computing device.

[0028] The memory 120 may be communicatively coupled to one or more processing devices of the image processing server 101 (e.g., one or more GPUs 110 and one or more CPUs 130). The optical flow accelerator 104 may operate in conjunction with the GPU 110 and / or CPU 130 or any combination thereof. The image processing server 101 may also include one or more input / output (I / O) components 134 for facilitating information exchange with various peripheral devices.

[0029] Despite Figure 1A Image processing server 101, application server 160, and client device 140 are shown as separate devices, but in various embodiments, any two (or all) of these devices may be combined on a single computing machine. For example, image processing server 101 and application server 160 may run on the same machine remotely accessed by client device 140. In another embodiment, image processing server 101, application server 160, and client device 140 may run on a user's (or developer's) computer (e.g., a desktop or laptop computer).

[0030] Figure 1BAn example computing device 102 according to at least one embodiment can perform optical flow processing through efficient and accurate processing of object boundary pixels. In some embodiments, computing device 102 may be an image processing server 101 or another computing device implementing optical flow accelerator 104. In some embodiments, optical flow accelerator 104 may combine one or more GPUs 110 for operation and may include an optical flow pyramid processing component 103 and a boundary pixel processing component 108 for performing accurate and efficient optical flow processing on individual pixels (and superpixels) (including boundary pixels (superpixels)) of image data 166. The operations performed by one or more GPUs 110 may include preprocessing (e.g., optical flow pyramid formation) and postprocessing of image data 166, the postprocessing including a feasibility check of the determined optical flow and the application of optical flow for generating additional images. In some embodiments, GPU 110 includes a plurality of cores 111, each core capable of executing a plurality of threads 112. Each core may run multiple threads 112 simultaneously (e.g., in parallel). In some embodiments, threads 112 may access registers 113. Register 113 may be a thread-specific register, with access to the register restricted to the corresponding thread. Furthermore, shared register 114 can be accessed by all threads of the core. In at least one embodiment, each core 111 may include a scheduler 115 for allocating computational tasks and processes among different threads 112 of the core 111. Dispatch unit 116 can implement the scheduled task on the appropriate thread using the correct private register 113 and shared register 114. Computing device 102 may include one or more input / output components 134 for facilitating information exchange with peripheral devices, as well as with users and developers.

[0031] In some embodiments, GPU 110 may have a (high-speed) cache 118, to which multiple cores 111 may share access. Additionally, computing device 102 may include GPU memory 119, where GPU 110 may store intermediate and / or final results (outputs) of various computations performed by GPU 110. After completing a specific task, GPU 110 (or CPU 130) may move the output to (main) memory 132. In some embodiments, CPU 130 may execute processes involving serial computation tasks, while GPU 110 may execute tasks suitable for parallel processing. In some embodiments, CPU 130 may determine which processes to execute on GPU 110 and which processes to execute on optical flow accelerator 104. In other embodiments, GPU 110 may determine which processes to execute on GPU 110 and which processes to execute on optical flow accelerator 104.

[0032] Figure 2This is an example pyramid optical flow architecture 200 according to at least one embodiment for performing efficient and accurate processing on object boundary pixels. The input to the optical flow may include image 202 and reference image 204, which can be any images captured at different times in a scene, depicting one or more moving (relative to the camera) objects. Image 202 and reference image 204 may undergo preprocessing involving pyramid generation 206. Additional preprocessing may include denoising, filtering, and other operations. Pyramid generation 206 may generate an image pyramid 208 comprising an arbitrary number of image layers of reduced size and resolution. Each subsequent image layer can be obtained by applying a kernel (e.g., Gaussian blur, binomial blur, or any other suitable kernel) to determine the pixel intensity of each image layer in image pyramid 208. Similarly, pyramid generation 206 may generate a reference image pyramid 210 based on reference image 204.

[0033] The image pyramid 208 and the reference image pyramid 210 can then be processed by the optical flow accelerator 104. The optical flow accelerator 104 can start from the top (lowest resolution) image layer and use motion cues from the lower resolution image layer to determine motion vectors for higher resolution layers, which are then used as cues for even higher resolution image layers. Cost quantity calculation 212 can be used to determine the motion vector for a given (super)pixel, which evaluates the mismatch between a (super)pixel in a neighborhood (search window) in the reference image layer identified by the cue motion vector and a (super)pixel in the same neighborhood in the corresponding resolution image layer. Boundary (super)pixel processing 214 can use multiple cues from the lower resolution image layer, combined as follows. Figure 4A -B describes this in more detail. Processing of each image layer may further include applying a semi-global matching (SGM) algorithm 216, which performs path cost calculation based on a cost amount 212. Each image layer may be further processed by a sub-pixel thinning algorithm 218, which calculates sub-pixel positions based on the winner's motion vector and the calculated cost. Each image layer may be further processed by a median filter 220, which performs smoothing on the determined motion vectors based on a weighted motion vector of a predetermined neighborhood of each (super)pixel.

[0034] As shown in box 222, all intermediate image layers of image pyramid 208 and reference image pyramid 210 can be processed iteratively (e.g., sequentially). The output of optical flow accelerator 104 may include a cost 224 associated with the layer 0 optical flow 226 of the determined highest-resolution zeroth image layer, where layer 0 optical flow 226 represents the motion vector of the original image 202 relative to the original reference image 204. Cost 224 and layer 0 optical flow 226 may undergo post-processing 230. Post-processing 230 may include a forward / backward check 232 that swaps image 202 and reference image 204 and determines the difference (e.g., Euclidean distance) between the forward and backward optical flows. If the difference is higher than a predetermined threshold, the obtained layer 0 optical flow 226 may be invalidated. Post-processing 230 may also include aperture filling 234 that fills any remaining gaps (e.g., caused by occlusion) in the determined optical flow. The optical flow output 240 of the post-processing 230 may include a final cost 242 and a motion vector 244, and may be used by the graphics application 162 and / or the image data generator 164 for any suitable purpose, such as generating additional images, tracking objects depicted in the images, etc.

[0035] Figure 3A The illustration depicts a pyramid structure 300 with multiple image layers of different resolutions for efficient image representation in optical flow processing, according to at least one embodiment. For simplicity and ease of explanation, Figure 3A Example image 302 with dimensions of 16×16 pixels is shown; however, it should be understood that an image or any size M0 x N0 can be represented similarly. Pixels of image 302 can be identified as two-dimensional vectors (x, y), where x and y take values ​​1, 2, ... 16, or any other interval of values ​​1, 2...M0 (or 1, 2...N0). For example, dark pixels of image 302 can be identified as pixels (8, 11). As used herein, the enumeration of pixels is illustrative only; in some embodiments, different enumerations may be used, for example, x (and y) taking values ​​0, 1, ... M0-1 (and 0, 1... N0-1). Each pixel of image 302 should be understood as a set of values ​​{I}(x, y) defined (and stored) in association with pixel coordinates. The set of values ​​{I} can include a single intensity value I (in the case of a black and white image) or multiple intensity values ​​(in the case of a color image), {I} = I1, I2, ..., which can be RGB (red, green, blue) intensity values, CMYK (cyan, magenta, yellow, tan) intensity values, or any other suitable intensity values. The term "pixel" should be understood as any unit of graphic information, which can be the smallest unit determined by the resolution of image 302. In some embodiments, a "pixel" can include multiple smallest units of graphic information considered to have the same motion vector.

[0036] like Figure 3AAs shown, image 302 is represented by a first image layer (IL) 304 and a second IL 306; image 302 may also be referred to herein as the zeroth IL. The first IL 304 may represent image 302 with a reduced resolution. For example, the first IL 304 may be a downscaled representation of image 302. In some embodiments, the size of the downscaled representation may be M1 x N1, where M1 = M0 / SF(1) and N1 = N0 / SF(1), where SF(1) is a first scaling factor. For example, the first IL 304 is a downscaled representation with a scaling factor SF(1) = 2, which performs uniform scaling along both dimensions of image 302. In some embodiments, different scaling factors may be used for (non-uniform) image scaling along different dimensions, for example, scaling factor SF(1) along the x-axis. X (1) = 4, scaling factor SF along the y-axis Y (1) = 2. Each pixel of the first IL 304 (also referred to as a superpixel in this paper) is larger than the area imaged by a single pixel of image 302 [SF(1)]. 2 (or in the case of non-uniform scaling, large (SF) X )x(SF Y This involves region imaging. Accordingly, each superpixel of the first IL 304 can be associated with multiple pixels of the image 302. For example, superpixel (4,6) of the first IL 304 can be associated with four pixels (7,11), (7,12), (8,11), and (8,12) of the image 302, as schematically depicted with corresponding dashed lines. Similarly, any superpixel (x,y) of the first IL 304 can be associated with pixels (2x-1,2y-1), (2x-1,2y), (2x,2y-1), and (2x,2y) of the image 302.

[0037] Each superpixel (x, y) of the first IL 304 may include a set of intensity values ​​{I} obtained from the intensity values ​​of the pixels in image 302. SP For example, the intensity I of the superpixel (x,y) of the first IL 304 SP (x,y) can be obtained by convolving the intensity of the pixels in image 302 with any suitable kernel K(x,y), such as I. SP (x,y)=∑ x′,y′ K(xx′;yy′)I P (x′, y′), such as binomial kernel, Gaussian kernel, etc. Kernel applications can include smoothing (blurring), denoising, filtering, or performing any other pixel processing operations.

[0038] Additional ILs can be formed similarly. For example, the second IL 306 can be a representation of the first IL 304 (and therefore image 302) at a coarser resolution. For example, the second IL 306 can be a scaled-down representation of the first IL 304. In some embodiments, the size of the second IL 306 can be M2 x N2, where M2 = M0 / SF(2) and N0 = N2 / SF(2), and SF(2) is a second scaling factor. In some embodiments, the second scaling factor can be the square of the first scaling factor, SF(2) = [SF(1)]. 2 For example, the depicted second IL 306 is a (uniformly) scaled-down representation of image 302 with a scaling factor of SF(2) = 4. In some embodiments, similar to the first IL 304, different scaling factors can be used for non-uniform image rescaling along different dimensions. Each superpixel of the second IL 306 depicts a region larger than [SF(2)] imaged by a single pixel of image 302. 2 The result is that each superpixel of the second IL 306 can be associated with multiple superpixels of the first IL 304. For example, a superpixel (2,3) of the second IL 306 can be associated with four superpixels (3,5), (3,6), (4,5), and (4,6) of the first IL 304, as schematically depicted with corresponding dashed lines. Similarly, any superpixel (x,y) of the second IL 306 can be associated with superpixels (2x-1,2y-1), (2x-1,2y), (2x,2y-1), and (2x,2y) of the first IL 304.

[0039] The intensity value of the superpixel of the second IL 306 can be obtained using the superpixel of the first IL 304, similar to the process described above for obtaining the superpixel of the first IL 304 from the pixels of image 302, for example, by smoothing, blurring, convolution, etc., on the superpixel of the second IL 306. In some embodiments, one or more intensity values ​​of the superpixel of the second IL 306 can be obtained directly from the intensity of the pixels of image 302.

[0040] Despite Figure 3AThe image depicts two ILs (excluding the zeroth IL, the image itself), but the pyramid structure 300 can include any number of ILs, including one IL, three ILs, four ILs, etc. In some embodiments, the number of ILs can be dynamic, for example, depending on the size of image 302, the resolution of image 302, the expectation of depicting a large number of fast-moving objects in image 302, etc. The number of ILs can also be application-dependent. For example, an image 302 expected to depict a larger number (e.g., three, four, etc.) of moving objects can be represented by a larger number of ILs, while an image 302 expected to depict a smaller number (e.g., one or two) of moving objects can be represented by a smaller number of ILs.

[0041] In some embodiments, pixels and superpixels can be associated with depth. In gaming applications, depth can be a discrete parameter indicating which objects are closer to the camera (advantage points) and which are farther away. In automotive, traffic monitoring, security, or any other real-world application, depth can be determined using radar data, lidar data, camera data (e.g., camera focal length data), and can be a continuous (or quasi-continuous) parameter. Superpixels can be similarly associated with depth, which can be calculated as the average depth of all pixels in image 302 or the superpixels of a first IL304 (second IL306, etc.) associated with a particular superpixel. In some embodiments, the depth of a superpixel can be determined as the depth of most pixels (or superpixels) associated with a particular superpixel, or determined in any other suitable manner.

[0042] Figure 3B The illustration depicts the use of a pyramid structure to identify object motion in optical flow processing according to at least one embodiment. For simplicity, Figure 3B The operations involving the first IL 304 and the zeroth IL (image 302) are illustrated, but it should be understood that they can be combined with the above text. Figure 3A Any pair of ILs (e.g., the (j+1)th IL and the jth IL) of the described pyramid structure 300 can perform a similar operation. The pyramid structure 300 can be used to identify motion vectors MV of at least some (and in some cases all) pixels of image 302. The motion vector MV(x,y) specifies the displacement of a pixel (x,y) relative to another reference image 308, which can be a previous image of the same scene or a modification (e.g., a temporal evolution) of the same scene. In some embodiments, the reference image 308 can be a future image of the same scene. For example, the motion vector of a pixel depicting a portion of a car door in image 302 can identify the position of the corresponding pixel (depicting the same car door) in a previous reference image 308.

[0043] In optical flow pyramid processing, pixel motion determination can begin with a higher IL (Internal Limit) of a coarser image with fewer superpixels and correspondingly representing the same scene, and then iteratively progresses to lower ILs until motion is identified for the final highest resolution image (zero IL). In this iterative pyramid processing, the motion vectors of the coarser ILs serve as "cues" to more accurately identify the motion vectors of the finer resolution ILs. In some embodiments, the topmost IL used in the pyramid processing may not have motion vector cues. In such embodiments, the motion vector of the topmost IL can be determined by segmenting the topmost IL of the image into superpixel blocks (e.g., 4x4 superpixel blocks) and matching the appearance of such superpixel blocks with the individual superpixel blocks of the topmost IL of a reference image, selecting the best match for each block. In some embodiments, the motion vector of the topmost IL can be provided by an application using optical flow image processing, such as a graphics application 162.

[0044] like Figure 3B As shown, determining the motion vector for a given pixel of image 302 (or similarly, for any superpixel of a higher j-th IL) may include: identifying the superpixel of the first IL 304 (or similarly, the superpixel of the j+1 IL) associated with the given pixel (or the superpixel of the j-th IL). For example, pyramid optical flow processing may first identify the superpixel (2,4) of image 302 associated with the superpixel of the first IL 304, as described above. Figure 3A As described. The motion vector for the superpixel (2,4) of the first IL 304, used as a cue for pixel (4,8) of image 302, can be MV1(x,y) = (4,1), which indicates the displacement of pixel (2,4) relative to the first IL (not shown) of reference image 308 by four superpixels along the x-axis and one superpixel along the y-axis. The motion vector MV1(x,y) can be used as a cue motion vector for identifying candidate motion vectors for associated pixels of image 302.

[0045] MV0(2x,2y)=2·MV1(x,y),

[0046] A scaling factor of 2 is used. In those embodiments using a different scaling factor SF(1), rescaling of the motion vector can be performed by replacing scaling factor 2 with an appropriate scaling factor SF(1). Due to the multiple pixels of image 302 (e.g., [SF(1)]), 2The same value MV0(2x,2y) is associated with a single superpixel of the first IL 304, so the same value MV0(2x,2y) can be used as a cue for multiple pixels, such as pixels (2x,2y), (2x-1,2y), (2x,2y-1), and (2x-1,2y-1) of image 302. As shown, the candidate motion vector MV0(4,8) = (8,2). Accordingly, pixel (12,10) (i.e., pixel (4,8) shifted by (8,2)) is used as the center pixel (dark pixel) of the search window 310 (shaded area) applied to the reference image 308. The center pixel (12,10) is used as the possible location of the pixel in the reference image 308 that matches pixel (4,8) in image 302. On the other hand, since image 302 (and reference image 308) have a higher resolution than the first IL 304 (and the corresponding IL of reference image 308), the actual location of the best matching pixel in reference image 308 may be different. To capture this possibility, the search window 310 can be selected to have a size L that is likely to capture the best matching pixel. x ×L y .like Figure 3B A 7×5 search window 310 is shown, but any other suitable search window may be used alternatively. The search window may be rectangular, square, or have some other shape. In some embodiments, the search window may be different (in size and shape) for the center pixel near the edge of image 302 and / or reference image 308. For example, the search window for the center pixel located near the edge or corner of reference image 308 may have a reduced (truncated) size, a triangular shape, etc.

[0047] The pixel intensity J of different pixels within the search window 310 of the reference image 308 i The corresponding pixel intensity J can be compared with the pixel intensity J of the pixel in image 302 (e.g., pixel (4,8) or any other pixel). i Comparisons can be made. Any suitable cost function (loss function) can be used to perform the comparison, for example, a simple quadratic cost function, C = ∑ i (I j -J i ) 2 It equally weights mismatches of different intensities (e.g., R, G, B intensities), or the weighted quadratic cost function C = ∑ i W i (I j -J i ) 2 Its usage experience (e.g., application-dependent) weight W iDifferent weights are applied to mismatches of varying intensities. Any other cost function can be used, including but not limited to mean absolute error, smoothed mean absolute error, log-cosh function, binary cross-entropy function, etc. Then, a pixel in reference image 308 that best matches (e.g., with the lowest cost function) a pixel in image 302 can be used to identify the motion vector of that pixel. For example, pixel (11, 10) in reference image 308 can be identified as the best match for pixel (4, 8) in image 302. Accordingly, Figure 3B The motion vector MV0(4,8)=(7,2) shown by the dashed arrow in the image can be selected as the final motion vector for pixel (4,8) in image 302.

[0048] In some embodiments, the motion vector for a given pixel of image 302 (and similarly for superpixels of each higher IL) can be determined based on more than a single pixel match. Figure 3C The illustration illustrates the use of a matching window to determine motion vectors during optical flow processing according to at least one embodiment. To avoid unintended false positive matches of pixels with similar intensities in image 302 and reference image 308, the geometric context of the pixels can be considered. More specifically, as depicted by the shaded square centered on pixel (4,8) of image 302, a neighborhood of the pixel—the matching window 302—can be selected. Although a 3×3 pixel matching window 302 is shown for illustration, any size matching window, including rectangular matching windows, cross-shaped matching windows, or any other suitable shape, can be used.

[0049] For each possible motion vector indicated by the dashed arrow, a corresponding matching window 305-n can be identified in reference image 308. For example... Figure 3C Two matching windows, 305-1 and 305-2, are shown, but L can be identified in reference image 308. x ×L y (Size of search window 310) Different matching windows. A cost function can be calculated for each matching window 305-n in the reference image 308, for example, by adding the cost function of each pixel in the matching window 303 of image 302 to the cost function of the corresponding pixel in the matching windows 305-n of the reference image 308. In some embodiments, the cost functions of different pixels can be added with uniform weights; for example, the cost function representing the mismatch of the center pixel can be weighted the same as the cost function representing the mismatch of the corner pixel. In some embodiments, the cost functions of different pixels can be weighted with different weights; for example, the cost function representing the mismatch of the center pixel can be weighted more than the cost function representing the mismatch of the corner or side pixels. Any suitable weight distribution of the cost functions can be used, for example, a weight distribution chosen empirically. Similar to Figure 3B You can choose the motion vector that connects the center of the matching window with the lowest cost function as the final motion vector for a given pixel of image 308 (e.g., pixel (4,8) in this example).

[0050] The above combination Figure 3A The technique described in -C can be similarly applied to identify motion vectors of the first IL based on motion vector cues from the second IL, to identify motion vectors of the second IL based on motion vector cues from the third IL, and so on, up to the last IL.

[0051] Figure 4A The illustration depicts pixel-superpixel association 400 during optical flow processing at object boundary pixels according to at least one embodiment. In a typical image depicting one or more moving objects and one or more stationary objects, some objects may be located in the foreground and closer to the camera (or observer), while others are located in the background and farther from the camera (or observer). The depiction of such objects is separated by edge pixels located near the visual boundaries between the objects. Figure 4A As shown, the depiction of object (vehicle) 402 includes portion 404, which comprises a white foreground (F) portion depicting a part of vehicle 402 and a shadowed background (B) portion depicting, for example, a stationary object (road surface) or a background vehicle. Boundary 406 separates the two portions.

[0052] Figure 4A The image 302 corresponding to part 404 is further depicted (from...). Figure 3A -C) pixelated region 408. Although region 408 is shown as having 16 pixels, this should be understood as illustrative only (not a scaling), as any number of pixels can be located near the edge region. Specifically, pixels with x-coordinates and y-coordinates in the interval [5,8] are shown. Region 408 of the image can be associated with the corresponding region 410 of the first IL 304, which has the x-coordinates and y-coordinates of the corresponding superpixel in the interval [3,8]. As described above... Figure 3A As described, the superpixel (3,2) of the first IL 304 is associated with four pixels of image 302: (5,3), (6,3), (5,4), and (6,4). Similarly, the superpixel (4,3) of the first IL 304 is associated with four pixels: (7,5), (8,5), (7,6), and (8,6).

[0053] For certain pixel-superpixel associations, it is clear whether a pixel and a superpixel belong to the foreground or background. For example, superpixel (4,2) and its associated pixels (7,3), (8,3), (7,4), and (8,4) depict only the foreground. Similarly, superpixel (3,3) and pixels (5,5), (6,5), (5,6), and (6,6) primarily depict the background, with only a small portion of pixel (6,5) capturing the foreground. On the other hand, superpixels (3,2) and (4,3) primarily depict the foreground, but their associated pixels (5,4) and (7,6) primarily depict the background.

[0054] Because the pixels of the image receive cue motion vectors from the superpixels of the first IL 304, pixels (5,4) and (7,6) may have large cue motion vectors, which will cause the corresponding search windows for these pixels to be centered at distant (and incorrect) locations within the reference image 308. This can cause optical flow processing to miss correct pixel matches and generate suboptimal optical flow. Figure 4A As shown, if edge pixels (5,4) and (7,6) were to receive cue motion vectors from superpixel (3,3), a more accurate search window would be obtained. To utilize the expanded pooling of motion vectors, optical flow processing can identify multiple cue motion vectors associated with superpixels that are distinct from those closely associated with a given edge pixel.

[0055] Figure 4B The illustration depicts the recognition of motion vector cues during optical flow processing of object boundary pixels according to at least one embodiment. As an example, Figure 4B This demonstrates the recognition of motion vector cues and the application of the recognized cues. Figure 4A The boundary (edge) pixels are (5,4), but other pixels can be processed similarly. For example... Figure 4B As shown, pixel (5,4) of image 302 is associated with superpixel (3,2) of first IL 304. Superpixel (3,2) may correspond to a moving background (vehicle) and may have MV1(3,2) = (3,4). Since pixel (5,4) of image 302 may depict a portion of a stationary background, in addition to cuing MV1(3,2), processing of pixel (5,4) may also involve the motion vectors of superpixels in a neighborhood of pixel (3,2), such as the motion vectors of the nearest neighbor superpixels in shadow region 403. In some embodiments, the neighborhood of superpixel (3,2) may include more than the nearest neighbor superpixels, for example, a second nearest neighbor superpixel or even more distant superpixels.

[0056] Each superpixel in the neighborhood can be associated with its own motion vector, and a different search window can be generated accordingly. For example, superpixels (3,2), (3,4), and (4,3) can have the same motion vector (3,4) for the first IL 304, and can correspond to the corresponding scaled-up candidate motion vector MV0 = (6,8) of image 302. This scaled-up motion vector can determine the search window 412 centered on pixel (11,12). Similarly, superpixel (3,3) can have different motion vectors, such as MV0 = (0,0) corresponding to the static background in this example (e.g., ...). Figure 4B (As shown by the cross in the diagram), and can lead to different search windows, for example, search window 414 centered on pixel (5,4). Therefore, optical flow processing will be able to determine the best match for pixel (5,4) and confirm that the final motion vector of that pixel is indeed MV0 = (0,0). Although only two different search windows appear in this example at the boundary between two objects (e.g., vehicle and background), in general, the described techniques can be used to identify any other number of search windows (e.g., up to the total number of different superpixels in the neighborhood, such as 9 in this example). In some cases, three (or even more) different motion vectors can be identified in a given superpixel neighborhood, for example, if the superpixel neighborhood overlaps with three objects moving relative to the camera (observer) at different speeds.

[0057] In some embodiments, the depth levels of individual pixels and superpixels can be known and used to optimize optical flow processing. For example, it may be known that the depth of pixel (5,4) in image 302 belongs to depth level D0. Furthermore, the depths of the individual neighboring superpixels of shadow region 403 can be identified. Some superpixels may have the same depth D0, while others may have different depths D1, D2, etc. In some embodiments, such as in gaming applications, superpixels with depths different from D0 can be excluded, and the motion vectors of excluded superpixels are not used as cues for pixel (5,4). In some embodiments, such as in practical applications where depth is a more continuous variable (e.g., determined from radar, camera, and / or lidar data), all superpixels with depth mismatches less than a certain (e.g., empirically determined) threshold can be considered. In such embodiments, superpixel depth mismatches can be added as another term to the cost function, such that larger depth mismatches are discouraged but not completely eliminated.

[0058] The above text combined Figure 4AThe technique described in -B can be similarly applied to identify the motion vectors of the boundary superpixels of the first IL based on the neighborhood cue from the second IL, to identify the motion vectors of the second IL based on the neighborhood cue from the third IL, and so on, until the last IL.

[0059] Although the above-described boundary pixel / superpixel processing techniques may depend on the identification of boundary pixels / superpixels, in some embodiments such identification is not performed because all pixels / superpixels can receive multiple cues from the coarser IL. For example, each (super)pixel in the j-th IL can receive several (e.g., nine) motion vector cues from the neighborhood of the (super)pixel in the (j-1)-th IL, regardless of whether the corresponding (super)pixel is located near the boundary between two (or more) objects.

[0060] Figure 5 and Figure 6 These are flowcharts of corresponding example methods 500 and 600 of pyramid optical flow processing with efficient identification and processing of object boundary pixels according to some embodiments of this disclosure. Methods 500 and 600 may be executed by one or more processing units, which may include (or communicate with) one or more memory devices. The one or more processing units may include an optical flow accelerator 104 implemented as one or more circuits. The optical flow accelerator may include (or communicate with) one or more high-speed memory buffers (e.g., external cue buffer 105) for storing cues for the top image layer, one or more motion vector buffers 107 for storing intermediate and final motion vectors identified during pyramid optical flow processing, and / or other memory buffers, registers, caches, etc. In at least one embodiment, methods 500 and 600 may be executed by a processing unit of computing device 102. In at least one embodiment, methods 500 and 600 may be executed by multiple processing threads (e.g., CPU threads and / or GPU threads), each thread performing the operation of one or more individual functions, routines, subroutines, or methods. In at least one embodiment, the processing threads implementing method 500 (and similarly, method 600) can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 500 (and similarly, method 600) can execute asynchronously relative to each other. The individual operations of methods 500 and 600 can be synchronized with... Figure 5 and Figure 6 The different sequences shown are executed. Some operations of these methods can be executed simultaneously with other operations. In at least one embodiment, Figure 5 and Figure 6 One or more of the operations shown may not always be performed.

[0061] Figure 5 This is a flowchart of an example method 500 for pyramid optical flow processing according to at least one embodiment, wherein object boundary pixels are identified and processed at the highest resolution level. Method 500 can be performed in video game processing, automotive applications, video content streaming, composite image / video generation, industrial / transportation / environmental / etc., surveillance contexts, and many other contexts. Method 500 can be used to obtain the optical flow of an image. The image can have multiple pixels of any suitable dimension, such as 720×480 pixels, 1280×720 pixels, 1920×1080 pixels, or can be an image of any other size. The image can be a black-and-white image or a color image. The image can be part of any video feed or a set of two or more images. The video feed or set of images may also include a reference image relative to which the optical flow in the image is determined. The optical flow can be any representation of the motion of an object depicted in the image relative to a depiction of the same object in the reference image. In some embodiments, the optical flow is represented as a map (field) of pixel displacements (motion vectors) MV(x,y) as a function of pixel positions in the image (or reference image). A motion vector can have multiple components associated with pixel displacements along different directions (spatial dimensions). In some embodiments, the motion vector can be specified in Cartesian coordinates, where x and y are coordinates along the respective Cartesian axes. The components of the motion vector (e.g., MV...) x MV y …) The corresponding displacement along the respective Cartesian axis can be specified. In some embodiments, any other set of coordinates (and motion vector components), such as polar coordinates, can be used. Although for the sake of brevity and simplicity, method 500 (and method 600) is described in terms of pixels, essentially similar techniques can be used to generate optical flow in a 3D image segmented into voxels. In such embodiments, the motion vector can have three components and can be defined in a 3D field, such as x, y, and z.

[0062] In box 510, the processing unit executing method 500 can obtain the first image layer (IL) of the image. The first image layer (e.g., Figure 3A (304) may include a first scaled representation of the image; the first scaled representation may be associated with a first scaling factor, for example, a scaling factor of 2 (e.g., Figure 3A (Medium), 4, or any other scaling factor. In box 520, method 500 may continue processing the device associating one pixel (e.g., any pixel) from a plurality of pixels of the image with a first superpixel of the first IL. For example, as Figure 4B As shown, pixel (5,4) is associated with superpixel (3,2) of the first IL 204.

[0063] In box 530, the processing unit executing method 500 can determine the motion vector of the first superpixel and the motion vectors of one or more neighboring superpixels. For example, as Figure 4B As shown, the motion vector of the first superpixel (3,2) can be MV1(3,2) = (3,4), and the motion vector of the neighboring superpixel (3,3) can be MV1(3,2) = (0,0). In some embodiments, one or more neighboring superpixels may include the nearest neighbor superpixel of the first superpixel, for example, the superpixel of the shadow region 403 centered on superpixel (3,2). Figure 5 As shown in the top-labeled section, determining the motion vectors of one or more neighboring superpixels and subsequently using the determined pixel to identify the motion vector of that pixel can be in response to identifying the first superpixel as an edge superpixel (box 532). For example, in some embodiments, to save computational costs, multiple superpixels can be reserved for identification (and subsequent use) for pixels (or superpixels) of the image that are boundary (edge) pixels. In some embodiments, identifying the first superpixel as a boundary superpixel may involve determining that the motion vector of the first superpixel differs from the motion vectors of at least some of the neighboring superpixels among one or more neighboring superpixels by at least a threshold. The threshold for the difference in motion vectors can be a pixel along any direction (e.g., (0,1) or (1,0)), or any other pre-selected threshold absolute value MV of the motion vectors. T For example, if For any neighboring superpixels of the first superpixel, the processing unit executing method 500 can classify the first superpixel as a boundary superpixel and identify the motion vectors of multiple neighbors (e.g., all nearest neighbors) of the first superpixel.

[0064] In some embodiments, the depth levels of individual pixels and superpixels can be known and used to optimize optical flow processing. For example, as shown in box 534, one or more neighboring superpixels (in box 540 below) selected for identifying the motion vector of a pixel can have the same depth level as that pixel. For example, neighboring superpixels with depth levels different from that pixel can be assumed to be unrelated to those pixels and excluded from further processing.

[0065] In box 540, method 500 can continue processing the unit to identify the motion vector of a pixel using the motion vector of the first superpixel and the motion vectors of one or more neighboring superpixels. For example, as Figure 5 The motion vectors of the identified pixels, as indicated by the bottom label, may include, in box 542, from the reference image (e.g., Figure 3B Select matching pixels from a set of pixels in the reference image 308 (e.g., Figure 3BPixels (12, 11) in the reference image. In some embodiments, this set of pixels in the reference image may include pixels of a search window associated with the motion vector of the first superpixel. For example, Figure 4B The pixels of the search window 412 can be associated with the motion vector (used as a cue) MV1 = (3,2) of the first superpixel. This motion vector can indicate the center of the search window 412 relative to that pixel (e.g., ...). Figure 4B The displacement of pixel (5,4) is an upscaled motion vector (using a first scaling factor, e.g., 2), for example, MV0 = 2·MV1 = (6,8). In some embodiments, one or more additional search windows (e.g., search window 414) may be identified, and the set of pixels may include pixels of one or more additional search windows. Each of the one or more additional search windows may be associated with a corresponding motion vector of one or more neighboring superpixels. For example, search window 414 may be associated with the motion vector MV1 = (0,0) of neighboring superpixel (3,3). Similar to the search window (e.g., search window 412) associated with the motion vector of the first superpixel, each of the one or more additional search windows may be centered at a position determined by the upscaled (with a first scaling factor) corresponding motion vector of one or more neighboring superpixels. In some embodiments, selecting matching pixels may include optimizing the neighborhood characterizing the pixel (e.g., such as...). Figure 3C The matching window 303 in the image 302 depicted in the image and the corresponding neighborhood of the matching pixel (e.g., Figure 3C The cost function of similarity in matching windows 305-1 or 305-2.

[0066] In box 550, method 500 may continue processing the unit to generate optical flow for the image. The optical flow of the image may include motion vectors identified for each of a plurality of pixels in the image (as described above in conjunction with boxes 520-540). It should be understood that the plurality of pixels in the image may include all pixels of the image, but may also include fewer than all pixels of the image. For example, some pixels may lack motion vectors, such as pixels depicting a new object or portions of a new object that are not present in the previously obtained reference image.

[0067] In box 560, method 500 may continue processing the unit to generate one or more additional images using the image and the optical flow generated for the image. For example, the processing unit may resolve the motion vectors of the individual pixels of the identified image into N parts (e.g., by dividing each motion vector into N equal or unequal parts), and use the obtained new motion vectors to generate additional images utilizing the intermediate (interpolated) positions of the corresponding pixels. Various operations using the generated optical flow may be facilitated by an API (e.g., an optical flow API) 122, which provides the user with access to the generated optical flow.

[0068] Figure 6 This is a flowchart of an example method 600 for pyramid optical flow processing according to at least one embodiment, wherein object boundary pixels are identified and processed at a reduced resolution level. Method 600 can be performed in conjunction with method 500 and can be used in the same applications and contexts as method 500. When more than one image layer is used to represent a given (highest resolution) image, the operations of method 600 can be performed in addition to those of method 500. For example, method 600 can use a first IL (referencing method 500) and can additionally use a second IL, a third IL, and so on. Each IL can include superpixels with progressively reduced resolution (increased coarsening), as described above in conjunction with... Figure 3A As described. In some embodiments, the final IL among a plurality of ILs may include a superpixel whose motion vector is provided as an input outside the image, for example, an application-specific motion vector, a user-provided motion vector, etc.

[0069] In box 610, method 600 may include: obtaining a second (or more typically, j-th) IL that includes a second (or more typically, j-th) scaled representation of the image. The second scaled representation may be associated with a second (or more typically, j-th) scaling factor. For example, the second (or more typically, j-th) scaling factor may be equal to S. 2 (or more generally, S) j ), where S is the first scaling factor.

[0070] In block 620, the processing unit performing method 600 may associate a first (or more typically, j-1) superpixel of the first IL with a second (or more typically, j) superpixel; the second (or more typically, j) superpixel may belong to the second (or more typically, j) IL.

[0071] In box 630, the processing unit performing method 600 can determine the motion vector of the second (or more generally, the j-th) superpixel and the motion vector of each of one or more of the neighboring superpixels of the second (or more generally, the j-th) superpixel.

[0072] In box 640, the processing unit performing method 600 can identify the motion vector of the first (or more typically, j-1) superpixel using the motion vector of the second (or more typically, j-th) superpixel and the motion vectors of each of one or more neighboring superpixels of the second (or more typically, j-th) superpixel. Boxes 610-640 can be repeated for all available ILs.

[0073] Figure 7 A block diagram depicts an example computer system 700 operating according to some embodiments of the present disclosure. In various illustrative examples, the example computer system 700 may be or includes... Figure 1A The image processing server 101, client device 140, and / or application server 160 shown herein are... Figure 1B The example computer system 700 is shown as computing device 102. It can be connected to other computer systems in a LAN, intranet, extranet, and / or the Internet. Computer system 700 can operate as a server in a client-server network environment. Computer system 700 can be a personal computer (PC), set-top box (STB), server, network router, switch, or bridge, or any device capable of executing a set of instructions (sequential or otherwise) specifying actions to be taken by that device. Furthermore, while only a single example computer system is illustrated, the term "computer" should also be understood to include any collection of computers that, individually or jointly, execute a set (or more) of instructions to perform any or more of the methods discussed herein.

[0074] Example computer system 700 may include processing device 702 (also called processor or CPU), main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM) etc.), static memory 706 (e.g., flash memory, static random access memory (SRAM) etc.) and auxiliary memory (e.g., data storage device 718), which can communicate with each other via bus 730.

[0075] Processing device 702 represents one or more general-purpose processing devices, such as microprocessors, central processing units, etc. More specifically, processing device 702 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing combinations of instruction sets. Processing device 702 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. According to one or more aspects of this disclosure, processing device 702 may be configured to execute instructions for implementing pyramid optical flow processing methods 500 and 600 for efficiently identifying and processing object boundary pixels.

[0076] The example computer system 700 may further include a network interface device 708 that can be communicatively coupled to a network 720. The example computer system 700 may further include a video display 710 (e.g., a liquid crystal display (LCD), a touch screen, or a cathode ray tube (CRT)), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), and a sound signal generation device 716 (e.g., a speaker).

[0077] Data storage device 718 may include a computer-readable storage medium (or more specifically, a non-transitory computer-readable storage medium) 728 thereon storing one or more sets of executable instructions 722. According to one or more aspects of this disclosure, the executable instructions 722 may include executable instructions for implementing pyramid optical flow processing methods 500 and 600 for efficiently identifying and processing object boundary pixels.

[0078] The executable instructions 722 may also reside wholly or at least partially within main memory 704 and / or processing device 702 during execution by the example computer system 700, which also constitute a computer-readable storage medium. The executable instructions 722 may also be transmitted or received over a network via network interface device 708.

[0079] Although the computer-readable storage medium 728 is Figure 7While the term "computer-readable storage medium" is used interchangeably with "single medium," it should be understood to include operations on a single medium or multiple media (e.g., centralized or distributed databases, and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of machine-executable instructions that cause the machine to perform any or more of the methods described herein. The term "computer-readable storage medium" should be understood to include, but is not limited to, solid-state memory and optical and magnetic media. For example, "memory" includes random access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM); ROM; magnetic or optical storage media; flash memory devices; energy storage devices; optical storage devices; acoustic storage devices; and any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable form (such as a computer).

[0080] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0081] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a” and “an”, “the”, and similar designations should be interpreted as encompassing both singular and plural forms, unless otherwise stated herein or obviously contradicted by the context, and not as a definition of a term. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0082] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0083] Unless otherwise stated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that are executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues). In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0084] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0085] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0086] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0087] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0088] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “operation,” “calculation,” “determine,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0089] Similarly, the term "processor" can refer to any device or part of memory that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or intermittently, or in parallel. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0090] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface (API). In some implementations, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In another implementation, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an API, or an inter-process communication mechanism.

[0091] While the discussion above illustrates example implementations of the described technologies, other architectures can be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for discussion purposes, various functions and responsibilities can be assigned and divided in different ways depending on the circumstances.

[0092] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method for obtaining optical flow in an image having multiple pixels, the method comprising: A first image layer IL is obtained, comprising a first scaled representation of the image, wherein the first scaled representation is associated with a first scaling factor; Associate one pixel of the plurality of pixels in the image with the first superpixel of the first IL; Determine the motion vector MV of the first superpixel and the MV of one or more neighboring superpixels of the first superpixel; The MV of the first superpixel and the MV of the one or more neighboring superpixels are used to identify the MV of the pixel in the image associated with the first superpixel; Generate the optical flow of the image, wherein the optical flow includes the MV of each of the plurality of pixels of the image; as well as One or more additional images are generated using the image and the optical flow of the image.

2. The method of claim 1, wherein using the MV of the one or more neighboring superpixels to identify the MV of the pixel is in response to identifying the first superpixel as an edge superpixel.

3. The method of claim 2, wherein identifying the first superpixel as an edge superpixel comprises: The MV of the first superpixel differs from the MV of at least some of the neighboring superpixels among the one or more neighboring superpixels by at least a threshold.

4. The method of claim 1, wherein one or more neighboring superpixels have the same depth level as the pixel.

5. The method of claim 1, wherein the one or more neighboring superpixels include a plurality of nearest neighbor superpixels of the first superpixel.

6. The method of claim 1, wherein identifying the MV of the pixel further comprises: Select matching pixels from a set of pixels in a reference image, wherein the set of pixels includes: The pixels within the search window associated with the MV of the first superpixel, and Pixels within one or more additional search windows, wherein each of the one or more additional search windows is associated with the corresponding MV of the one or more neighboring superpixels.

7. The method of claim 6, wherein each of the one or more additional search windows is centered at a position determined by the corresponding MV of the one or more neighboring superpixels magnified using the first scaling factor.

8. The method of claim 6, wherein selecting the matching pixel comprises: Optimize the cost function that characterizes the similarity between the neighborhood of the pixel and the corresponding neighborhood of the matching pixel.

9. The method according to claim 1, further comprising: Obtain a second IL, the second IL including a second scaled representation of the image, wherein the second scaled representation is associated with a second scaling factor; Associating the first superpixel of the first IL with a second superpixel, wherein the second superpixel belongs to the second IL; Determine the MV of the second superpixel and the MV of each of the one or more neighboring superpixels of the second superpixel; as well as The MV of the first superpixel is identified using the MV of the second superpixel and the MV of each of the one or more neighboring superpixels of the second superpixel.

10. The method of claim 9, wherein the first IL and the second IL are ILs of a plurality of ILs of the image, and wherein the final IL of the plurality of ILs includes a superpixel whose MV is provided as an input outside the image.

11. An optical flow accelerator, comprising: One or more circuits, said one or more circuits being used for: A first image layer IL is obtained, the first IL including a first scaled representation of an image having a plurality of pixels, wherein the first scaled representation of the image is associated with a first scaling factor; Associate one pixel of the plurality of pixels in the image with the first superpixel of the first IL; as well as Determine the motion vector MV of the first superpixel and the MV of one or more neighboring superpixels of the first superpixel; and One or more memory buffers, said one or more memory buffers being used for: Store the determined MV. The one or more circuits mentioned above are also used for: The MV of the first superpixel and the MV of the one or more neighboring superpixels are used to identify the MV of the pixel in the image associated with the first superpixel; Generate the optical flow of the image, wherein the optical flow includes the MV of each of the plurality of pixels of the image; as well as One or more additional images are generated using the image and the optical flow of the image.

12. The optical flow accelerator of claim 11, wherein the one or more circuits are used for: The first superpixel is identified as an edge superpixel based on the fact that the MV of the first superpixel differs from the MV of at least some of the neighboring superpixels among the one or more neighboring superpixels by at least a threshold.

13. The optical flow accelerator of claim 11, wherein the one or more neighboring superpixels include a plurality of nearest neighbor superpixels of the first superpixel.

14. The optical flow accelerator of claim 11, wherein, in order to identify the MV of the pixel, the one or more circuits are configured to: Select matching pixels from a set of pixels in a reference image, wherein the set of pixels includes: The pixels within the search window associated with the MV of the first superpixel, and Pixels within one or more additional search windows, wherein each of the one or more additional search windows is associated with the corresponding MV of the one or more neighboring superpixels.

15. The optical flow accelerator of claim 11, wherein the one or more circuits are further configured to: Obtain a second IL, the second IL including a second scaled representation of the image, wherein the second scaled representation is associated with a second scaling factor; Associating the first superpixel of the first IL with a second superpixel, wherein the second superpixel belongs to the second IL; Determine the MV of the second superpixel and the MV of each of the one or more neighboring superpixels of the second superpixel; as well as The MV of the first superpixel is identified using the MV of the second superpixel and the MV of each of the one or more neighboring superpixels of the second superpixel; and The one or more memory buffers mentioned therein are also used for: Store the MV of the second superpixel and the MV of each of the one or more neighboring superpixels of the second superpixel.

16. The optical flow accelerator of claim 15, wherein the first IL and the second IL are ILs of a plurality of ILs of the image, and wherein the final IL of the plurality of ILs includes a superpixel whose MV is provided as an input outside the image.

17. A system comprising: Memory devices; as well as Processing device, communicatively coupled to the memory device, the processing device being used for: A first image layer IL is obtained, the first IL including a first scaled representation of an image having a plurality of pixels, wherein the first scaled representation of the image is associated with a first scaling factor; Associate one pixel of the plurality of pixels in the image with the first superpixel of the first IL; Determine the motion vector MV of the first superpixel and the MV of one or more neighboring superpixels of the first superpixel; The MV of the first superpixel and the MV of the one or more neighboring superpixels are used to identify the MV of the pixel in the image associated with the first superpixel; Generate the optical flow of the image, wherein the optical flow includes the MV of each of the plurality of pixels of the image; as well as One or more additional images are generated using the image and the optical flow of the image.

18. The system of claim 17, wherein the processing device is further configured to: The generated optical flow is provided to the user via an application programming interface.

19. The system of claim 17, wherein the processing device is further configured to: The first superpixel is identified as an edge superpixel based on the fact that the MV of the first superpixel differs from the MV of at least some of the neighboring superpixels among the one or more neighboring superpixels by at least a threshold.

20. The system of claim 17, wherein the processing device is further configured to: Obtain a second IL, the second IL including a second scaled representation of the image, wherein the second scaled representation is associated with a second scaling factor; Associating the first superpixel of the first IL with a second superpixel, wherein the second superpixel belongs to the second IL; Determine the MV of the second superpixel and the MV of each of the one or more neighboring superpixels of the second superpixel; as well as The MV of the first superpixel is identified using the MV of the second superpixel and the MV of each of the one or more neighboring superpixels of the second superpixel.

Images