Method and electronic device for managing latency in a partial-frame-rendering

Abstract

A method includes identifying one or more moving objects in a real-world scene being obtained; obtaining, the movement of the one or more moving objects; and determining, a buffering rate for rendering obtained frames of the real-world scene, corresponding to the obtained movement of the one or more moving objects.

Description

METHOD AND ELECTRONIC DEVICE FOR MANAGING LATENCY IN A PARTIAL-FRAME-RENDERING

[0001] The present disclosure generally relates to the field of display devices such as Video See Through (VST) devices and more particularly, the present disclosure relates to a method and a system for latency management for partial frame rendering in such VST devices.

[0002] Generally, images and videos are preferable sources for users to consume content. The images and videos assist users in learning and understanding different types of content, for example, working on any components, concepts, etc. The video is recorded and rendered by a device, for example, a mobile, a video camera, etc. Viewing experience via display devices such as a mobile phone, laptop, LED display devices, and the like, is generally restricted to a 2-Dimensional (2-D) space.

[0003] A Video See Through (VST) device is an electronic display device that allows the user to see what is shown on the screen while still being able to see through the screen. Examples of VST devices include head-up displays, augmented reality systems, and the like. The VST device may be a Head Mounted Display (HMD) device. The VST device may be mounted on a user's forehead covering the eyes of the user. The VST device includes at least one display screen (digital screen) between the real world and the eyes of the user. The at least one screen may include a see-through screen and may be placed very close to the eyes of the user.

[0004] This summary is provided to introduce a selection of concepts, in a simplified format, that are further described in the detailed description of the disclosure. This summary is neither intended to identify key or essential inventive concepts of the invention nor is it intended to determine the scope of the disclosure.

[0005] According to an embodiment of the disclosure, a method may be disclosed. In an embodiment, the method may include identifying one or more moving objects in a real-world scene being obtained. The method may include obtaining, continuously, the movement of the one or more moving objects. The method may include determining, a buffering rate for rendering obtained frames of the real-world scene, corresponding to the obtained movement of the one or more moving objects.

[0006] According to an embodiment of the disclosure, a VST device may be disclosed. In an embodiment, the VST device may include a camera. The VST device may include at least one processor comprising processing circuitry. The VST device may include at least one memory including one or more instructions. In an embodiment, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to identify one or more moving objects in a real-world scene being obtained. In an embodiment, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to obtain, the movement of the one or more moving objects. In an embodiment, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to determine, a buffering rate for rendering obtained frames of the real-world scene, corresponding to the obtained movement of the one or more moving objects

[0007] According to an embodiment of the disclosure, a computer-readable medium containing instructions is disclosed. The instructions, when executed by at least one processor, cause the VST device to perform the method provided.

[0008] To further clarify the advantages and features of the present disclosure, a more particular description of the disclosure will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawing. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting its scope. The disclosure will be described and explained with additional specificity and detail with the accompanying drawings.

[0009] These and other features, aspects, and advantages of the present disclosure will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

[0010] Figure 1 illustrates a scenario depicting a real-world scene being captured using a Video See Through (VST) device, in the related art;

[0011] Figure 2A illustrates a process flow for a full frame delivery mechanism, in accordance with the related art;

[0012] Figure 2B illustrates a process flow for a partial frame delivery mechanism, in accordance with the related art;

[0013] Figure 2C is a pictorial depiction of a screen tearing, in accordance with the related art;

[0014] Figure 2D is a graph illustrating an exemplary relationship between latency and fill-percentage, in accordance with the related art;

[0015] Figure 3 illustrates an environment comprising a VST device for managing latency in a partial-frame-rendering VST device while rendering the real-world scene being captured by the VST device, in accordance with an embodiment of the present disclosure;

[0016] Figure 4 illustrates the device for managing latency while rendering the real-world scene being captured by the device, in accordance with an embodiment of the present disclosure;

[0017] Figure 5 illustrates a process flow of the device for managing latency while rendering the real-world scene, in accordance with an embodiment of the present disclosure;

[0018] Figure 6A illustrates a process flow for the generation of the velocity map and the working of a measurement module , in accordance with an embodiment of the present disclosure;

[0019] Figures 6B-6C illustrate the working of the measurement module with reference to a foreground object and a background object, in accordance with an embodiment of the present disclosure;

[0020] Figure 6B, in particular, illustrates a difference of movement as captured by the device between the foreground object and the background object, in accordance with an embodiment of the present disclosure;

[0021] Figures 7A-7B illustrate the working of a buffering rate module with reference to a velocity map, in accordance with an embodiment of the present disclosure;

[0022] Figure 7A illustrates an exemplary set-up for determination of a predetermined threshold velocity value, in accordance with an embodiment of the present disclosure;

[0023] Figure 7B illustrates a process flow for determination of a movement score, in accordance with an embodiment of the present disclosure;

[0024] Figure 8A illustrates the device for dynamically optimizing latency for rendering the real-world scene in the device, in an alternative embodiment of the present disclosure;

[0025] Figure 8B illustrates a comparison of the utilization of the processing resources while rendering the real-world scene in the device, in accordance with an embodiment of the present disclosure;

[0026] Figures 9A-9B illustrate the working of a renderer of the device, in accordance with an embodiment of the present disclosure;

[0027] Figure 10 illustrates a process flow for modifying a default partial-frame rendering delivery-policy of the device, in accordance with an embodiment of the present disclosure;

[0028] Figure 11 is a flowchart illustrating a method for managing latency in the device, in accordance with an embodiment of the present disclosure; and

[0029] Figure 12 is a flowchart illustrating a method for dynamically optimizing latency for rendering the scene in the device, in accordance with an embodiment of the present disclosure.

[0030] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent operations involved to help to improve understanding of aspects of the present disclosure. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0031] For the purpose of promoting an understanding of the principles of the disclosure, reference will now be made to the various embodiments and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the disclosure as illustrated therein being contemplated as would normally occur to one skilled in the art to which the disclosure relates.

[0032] The term "some" or "one or more" as used herein is defined as "one", "more than one", or "all." Accordingly, the terms "more than one," "one or more" or "all" would all fall under the definition of "some" or "one or more". The term "an embodiment", "another embodiment", "some embodiments", or "in one or more embodiments" may refer to one embodiment or several embodiments, or all embodiments. Accordingly, the term "some embodiments" is defined as meaning "one embodiment, or more than one embodiment, or all embodiments."

[0033] The terminology and structure employed herein are for describing, teaching, and illuminating some embodiments and their specific features and elements and do not limit, restrict, or reduce the spirit and scope of the claims or their equivalents. The phrase "exemplary" may refer to an example.

[0034] More specifically, any terms used herein such as but not limited to "includes," "comprises," "has," "consists," "have" and grammatical variants thereof do not specify an exact limitation or restriction and certainly do not exclude the possible addition of one or more features or elements, unless otherwise stated, and must not be taken to exclude the possible removal of one or more of the listed features and elements, unless otherwise stated with the limiting language "mush comprise" or "needs to include".

[0035] Whether or not a certain feature or element was limited to being used only once, either way, it may still be referred to as "one or more features", "one or more elements", "at least one feature", or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" feature or element does not preclude there being none of that feature or element unless otherwise specified by limiting language such as "there needs to be one or more" or "one or more element is required."

[0036] It should be appreciated that the blocks in each flowchart and combinations of the flowcharts may be performed by one or more computer programs which include computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory or the one or more computer programs may be divided with different portions stored in different multiple memories.

[0037] Any of the functions or operations described herein can be processed by one processor or a combination of processors. The one processor or the combination of processors is circuitry performing processing and includes circuitry like an application processor (AP), a communication processor (CP), a graphical processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

[0038] The processor may include various processing circuitry and / or multiple processors.　 For example, as used herein, including the claims, the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of at least one processor, individually and / or collectively in a distributed manner, may be configured to perform various functions described herein. As used herein, when "a processor", "at least one processor", and "one or more processors" are described as being configured to perform numerous functions, these terms cover situations, for example and without limitation, in which one processor performs some of recited functions and another processor(s) performs other of recited functions, and also situations in which a single processor may perform all recited functions. Additionally, the at least one processor may include a combination of processors performing various of the recited / disclosed functions, e.g., in a distributed manner.　 At least one processor may execute program instructions to achieve or perform various functions.

[0039] Unless otherwise defined, all terms, and especially any technical and / or scientific terms, used herein may be taken to have the same meaning as commonly understood by one having ordinary skill in the art.

[0040] Figure 1 illustrates a scenario depicting a real-world scene 100S being obtained (e.g. captured) using a Video See Through (VST) device 150, in the related art. The real-world scene 100S may be obtained (e.g. captured) in the form of images or series of images, and the like and rendered on a screen of the device 150. The images 110V may not be of optimal quality and may have a tearing effect as shown in the example image of Figure 2C.

[0041] The VST device 150 gives viewers a more immersive viewing experience via a pass-through mode of the VST device 150. In the pass-through mode, the user is able to see the real world in real time while wearing the VST device 150. For a delightful user experience, the pass-through mode of the VST device 150 may be able to mimic the pair of human eyes as closely as possible. To realize the pass-through mode, the VST device 150 has a transparent display and includes a pair of cameras (depicting each eye of the pair of eyes of a human being). The two cameras obtain (e.g. capture) a scene of the real-world and project the scene on the transparent display (digital screen) of the VST device 150 in real-time.

[0042] The pass-through mode of the VST device 150 may be enabled in various scenarios such as a Mixed Reality scenario. In the mixed reality scenario, the attention of the user is more focused on the virtual content. The pass-through mode may be enabled during an Augmented Reality (AR) scenario, wherein the user has his full attention on the AR content. The pass-through mode may be enabled during an Extended Reality (XR) scenario. In any scenario, it is desired that the latency should be minimal. The passthrough experience of the VST device has to be as seamless as possible to the user so that the user may never feel the presence of an opaque screen blocking the physical world.

[0043] Accordingly, it is desired that the latency induced in the VST device is as low as possible. Latency includes the time elapsed from the moment of obtaining (e.g. capturing) of the frames of the real-world scene by a camera sensor of the VST device to the moment of rendering of the obtained (e.g. captured) frames on the screen. The induced latency may be referred to as Photon-To-Photon (P2P) latency.

[0044] The issue of P2P latency may be addressed by using a frame delivery mechanism known as a partial-frame delivery. However, the partial frame-delivery has certain shortcomings. A parameter associated with the partial frame delivery is called a partial frame fill percentage. The partial frame fill percentage indicates a wait time for a processor (e.g. Graphics Processing Unit (GPU)). The partial frame fill percentage is the amount of percentage of the frame required to be filled by the camera buffer before the frame can be processed and sent to the next stage.

[0045] Figure 2A, of the related art, illustrates a process flow 200A for a full frame delivery mechanism, in accordance with the related art. A passthrough service 210 waits for the whole of a camera buffer to be filled completely and then transfers the passthrough to a renderer 220 to render on a screen of the device. A camera sensor 150 is used to take the exposure data which is used to update the camera buffer. Further, the camera buffer 230 including the camera frame information files, fills using the data received from the camera sensor 150. The passthrough service 210 waits for the camera buffer to fill completely and then finally accesses the camera buffer information. Thereupon, the renderer 220 renders the frame received from the passthrough service 210 onto the screen.

[0046] Figure 2B, of the related art, illustrates a process flow 200B for a partial frame delivery mechanism, in accordance with the related art. The passthrough service 210 does not wait for the whole of the camera buffer 230 to be filled. Instead, the passthrough service 210 transfers the partially filled buffer details to the renderer to render on the screen resulting in less P2P latency. The camera sensor 150 is used to take the exposure data which is used to update the camera buffer. Further, the camera buffer containing the camera frame information files, is filled using the data received from the camera sensor. Moreover, the passthrough service 210 waits for the camera buffer to fill partially and simultaneously accesses the camera buffer information. Furthermore, the renderer renders the frame received from the passthrough service onto the screen.

[0047] Further, conventionally, the partial frame fill percentage remains constant for a configuration of the VST device or at least for a certain usage configuration of the VST device. As a result, the static partial frame delivery mechanism leads to a static utilization of the CPU / GPU resources for a given scene being rendered. Further, this utilization is irrespective of the real-world scene being dynamic or static. There is an opportunity to make the changes dynamically so that the GPU and CPU save their computations depending upon the scene.

[0048] Additionally, a problem associated with partial frame delivery is screen tearing, which is caused due to the excessive motion in the scene. The screen tearing is solved by increasing the partial frame fill percentage to a higher value (static). However, the increase in the partial frame fill percentage wastes the GPU resources when not many changes are there in the scene.

[0049] If the device or objects in the environment are in motion and the partial fill percentage is less and not sufficient, then there may be a possibility that the scene being captured changes significantly between subsequent frames. Hence, screen tearing may be observed. Figure 2C is a pictorial depiction of a screen tearing, in accordance with the related art.

[0050] Figure 2D is a graph illustrating an exemplary relationship between latency (in milliseconds) and partial frame fill-percentage, in accordance with the related art. A balance between the fill-percentage and the latency is desired. Screen tearing is typically observed when the fill percentage is less (typically <60%). However, increasing the fill percentage dissolves the purpose of using partial frame delivery as it increases the latency.

[0051] Further, the partial frame fill percentage, by convention is set to be a constant. As a result, the load on the resources of the VST device is also the same irrespective of the nature of the scene - stable or dynamic. This leads to a loss of opportunity to save the GPU usage according to the variable dynamism of the scene. In some situations, the GPU is required to wait for a longer period of time if the current scene is very different from the previous scene. Whereas, if the scene is stable, then the GPU has waits for less time and sends the frame immediately for rendering.

[0052] Therefore, in view of the above-mentioned problems, it is advantageous to provide an improved device, system and method that can overcome the above-mentioned problems and limitations associated with the partial frame delivery in the VST devices.

[0053] Figure 3 illustrates a device 150 for managing latency in a partial-frame-rendering Video See Through (VST) device 150 (interchangeably referred to herein as the device 150) while rendering a real-world scene 100S being obtained (e.g. captured) by the device 150, in accordance with an embodiment of the present disclosure. The real-world scene 100S may be obtained (e.g. captured) in the form of images or series of images, and the like and rendered on a screen 352 of the device 150. In an embodiment, the method disclosed here, which is performed by the device 150, may be performed by a system, which is communicatively coupled with the device 150. In an embodiment, one or more components disclosed here, which are included in the device 150, may be included in the system, which is communicatively coupled with the device 150. The system may be communicatively coupled with the device 150 for rendering the scene 100S in the form of images or videos.

[0054] The device 150 may include camera 802, at least one processor 804 including processing circuitry and at least one memory 808 including one or more instructions. At least one memory 808 including one or more instructions, executed by the at least one processor 804 individually or collectively, to cause the device 150 to perform any one of method described in the present disclosure. The scene 100S may be in a form of images or videos.

[0055] In an embodiment, the device 150 may be a smartphone, a camera, or any other electronic device using a partial frame delivery mechanism having one or more cameras compatible with capturing or recording images, video, etc. of the scene 100S (the real-world scene), without departing from the scope of the present disclosure. In an embodiment, the device 150 employs the partial frame delivery mechanism. In an exemplary embodiment, a preview frame latency in the device 150 such as a smart phone may be at 120 milli seconds. The device 310 may be able to reduce the preview frame latency using the partial frame delivery mechanism. Similarly, the device 150 may include line scan-cameras employing the partial frame deliver mechanism or similar mechanisms.

[0056] In an embodiment, the device 150 may include multiple layers, for example, an application layer, a file system layer, etc. The application layer may include a video player application, a gallery application, or a camera application, without departing from the scope of the present disclosure. Further, the file system layer may include a file reader, a CoDec, a frame data, and a file writer. The file reader may be configured to read a video recorded by the application layer. The CoDec detects / checks the format of the recorded video (file) and also checks the coder-decoder part of the format of the file. Further, the frame data is prepared / formed by the CoDec for rendering a plurality of frames associated with the video on the display of the device 150.

[0057] In an embodiment, the device 150 includes at least one processor 804, at least one memory 808 and a camera 802. The at least one processor 804 may be disposed in communication with a communication network via a network interface. In an embodiment, the network interface may be the I / O interface. The network interface may connect to the communication network to enable the communication of the device 150. The network interface may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 702.11a / b / g / n / x, etc. The communication network may include, without limitation, a direct interconnection, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, etc. Using the network interface and the communication network, the device 150 may communicate with other devices. The network interface may employ connection protocols including, but not limited to, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 702.11a / b / g / n / x, etc.

[0058] In an embodiment, the at least one memory 808 may be communicatively coupled to the at least one processor 804. The at least one memory 808 may be configured to store data, and instructions executable by the at least one processor 804. In one embodiment, the at least one memory 808 may be provided within the device 150. In an embodiment, the at least one memory 808 may be provided being remote from the device 150. In an embodiment, the at least one memory 808 may communicate with the at least one processor 804 via a bus. In an embodiment, the at least one memory 808 may be located remotely from the at least one processor 804 and may be in communication with the processor 804 via a network. The at least one memory 808 may include, but is not limited to, a non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media including, but not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like.

[0059] In one example, the at least one memory 808 may include a cache or random-access memory for the processor 804. In an example, the at least one memory 808 is separate from the at least one processor 804, such as a cache memory of a processor, the system memory, or other memory. The at least one memory 808 may be an external storage device or database for storing data. The at least one memory 808 may be operable to store instructions executable by the at least one processor 804. The functions, acts, or tasks illustrated in the figures or described may be performed by the programmed at least one processor 804 for executing the instructions stored in the memory 808. The functions, acts, or tasks are independent of the particular type of instruction set, storage media, processor, or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro-code, and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, and the like.

[0060] In an embodiment, the plurality of modules 400 described below may be included within the memory 808. The plurality of modules 400 may include a set of instructions that may be executed to cause the system 310, in particular, the processor 804 of the system 310, to perform any one or more of the methods / processes disclosed herein. The plurality of modules 400 may be configured to perform the operations of the present disclosure using the data stored in the database. For instance, the plurality of modules 400 may be configured to perform the operations disclosed in Figures 11-12.

[0061] In an embodiment, each of the plurality of modules 400 may be a hardware unit which may be outside the memory 808. Further, the memory 808 may include an operating system for performing one or more tasks of the system, as performed by a generic operating system. Each of the modules 400 may be in communication with one another and the processor 804.

[0062] Figure 4 illustrates the device 150 for managing latency in the device 150 while rendering the real-world scene 100S being captured by the device 150, in accordance with an embodiment of the present disclosure. In an embodiment, the method disclosed here, which is performed by the device 150, may be performed by a system, which is communicatively coupled with the device 150. In an embodiment, one or more components disclosed here, which are included in the device 150, may be included in the system, which is communicatively coupled with the device 150.

[0063] The device 150 (or, the system which is communicatively coupled with the device 150) includes a plurality of modules 400 including a detection module 410, a measurement module 420, and a buffering rate module 430. The detection module 410 is configured for detecting for moving objects in the scene 100S being obtained (e.g. captured). The measurement module 420 is configured for measuring, continuously, the movement of one or more moving objects in the scene 100S. The buffer rate module 430 is configured for determining (e.g. modifying, varying), dynamically, a buffering rate for rendering captured frames of the real-world scene 100S. The buffer rate module 430 is configured for varying the buffering rate in proportion to the measured movement of the one or more moving objects.

[0064] The working and functioning of the plurality of modules 400 of the device 150 (or, the system which is communicatively coupled with the device 150) have been described in detail with reference to the following Figures.

[0065] Figure 5 illustrates a process flow 500 of the device 150 for managing latency in the device 150 while rendering the real-world scene 100S, in accordance with an embodiment of the present disclosure. In an embodiment, the method disclosed here, which is performed by the device 150, may be performed by a system, which is communicatively coupled with the device 150. In an embodiment, one or more components disclosed here, which are included in the device 150, may be included in the system, which is communicatively coupled with the device 150.

[0066] In an embodiment, the device 150 (or, the system which is communicatively coupled with the device 150) obtains (e.g. captures) the scene 100S, where the camera sensor records the exposure data and updates the camera buffer 530. The camera buffer 530 may temporarily hold frame information before further processing.

[0067] The passthrough service 540 may wait until a predetermined (e.g. defined) percentage of the camera buffer is filled before accessing and transmitting the frames for further processing. This waiting mechanism ensures that an adequate amount of motion data is available before processing begins, maintaining frame continuity and improving motion analysis accuracy.

[0068] Once the buffer reaches the desired fill percentage, the passthrough service 540 transmits the frames into the renderer 550 for further rendering and visualization.

[0069] In an embodiment, upon detection of moving objects by the detection module 410 in the scene 100S, the measurement module 420 is further configured for generating a velocity map 510 associated with the moving objects. The velocity map 510 comprises velocity values of moving pixels associated with the one or more moving objects in captured frames (520N, 520N-1, 520N-2). The frame 520N is the last frame captured by the device 150, the frame 520N-1 is the frame obtained (for example, captured) just before the frame 520N and frame 520N-2, the frame just before the frame 520N-1.

[0070] With reference to the scene 100S, various scenarios may emerge. In an exemplary scenario, the device 150 may be stationary and an object in the scene 100S may be moving. In an exemplary scenario, the device 150 may be moving and the object in the scene 100S may be stationary. In an exemplary scenario, both the device 150 and the object in the scene 100S may be moving. In an exemplary scenario, both the device 150 and the object in the scene 100S may be stationary. In all these probable scenarios, the device 150 will be effective as it is based upon the movement of the objects in the scene 100S with respect to the frames (520N, 520N-1, 520N-2) being captured.

[0071] In an embodiment, the measurement module 420 is configured for identifying at least one foreground object such as the foreground object 510F from among the one or more objects in the scene 100S by performing a segmentation of the captured frames (520N, 520N-1, 520N-2). The measurement module 420 is configured for identifying (for example, segmenting) that the foreground object 510F is the foreground object in the scene 100S and other objects such as the background object 610B are not the foreground object.

[0072] In an embodiment, the measurement module 420 is configured for correlating a displacement of the pixel movement between the first image frame 520N-2 and the second image frame 520N-1, with the time elapse between the first image frame 520N-2 and the second image frame 520N-1. The time elapsed may be correlated from the rate of frames (520N, 520N-1, 520N-2) rendered per unit time and may be expressed as fps (frames per second) 560. In an embodiment, the device 150 may further obtain (e.g. generate) the optical flow-map 520 for each of the identified foreground objects 510F. The optical flow-map 520 comprises displacement values of the moving pixels of the identified foreground objects 510F in the obtained (e.g. captured) frames (520N, 520N-1, 520N-2). The velocity map 510 is obtained (e.g. generated) by correlating the displacement in the optical flow-map 520 and a time duration elapsed between the obtained (e.g. captured) frames (520N, 520N-1, 520N-2). The velocity map 510 and the optical flow-map 520 will be further explained below. Figure 6A illustrates a process flow 600 for the generation of the velocity map 510 and the working of the measurement module 420, in accordance with an embodiment of the present disclosure. The velocity map 510 indicates a complexity of the scene 100S. With reference to the scene 100S, the velocity map 510 may be indicative of the velocity of each pixel in the frames (520N, 520N-1, 520N-2). The detection module 510 may detect foreground of the obtained frames based on the previous two frames (520N-1, 520N-2) The measurement module may generate dense optical flow map. The measurement module 420 may generate velocity map 510 of the frame for correlating a displacement of the pixel movement between the first image frame 520N-2 and the second image frame 520N-1, with fps 560. The velocity map 510 of the frame may indicate scene complexity.

[0073] Figures 6B-6C illustrate the working of the measurement module 420 with reference to a foreground object 510F and a background object 610B, in accordance with an embodiment of the present disclosure.

[0074] Figure 6B, in particular, illustrates a difference of movement as captured by the device 150 between the foreground object 510F and the background object 610B, in accordance with an embodiment of the present disclosure. It may be appreciated that due to a projective geometry of the device 150, the movements in the foreground are much more pronounced as compared to the movements in the background of the scene 100S being captured.

[0075] For the same movement of objects in the real world, the motion fields for the foreground objects 510F in the foreground of the scene 100S are much more as compared to those in the background. Hence, the measurement module 420 is configured for segmenting out the foreground object 510F from the captured frames (520N, 520N-1, 520N-2). The measurement module 420 is further configured for generating the velocity map 510 for the identified foreground object 510F.

[0076] As illustrated in Figure 6B, the top view diagram 630 provides a schematic representation of how objects within the field of view of the device 150 are positioned relative to the principal axis. In the top view diagram 630, the foreground object 510F and the background object 610B are shown at different depths from the camera. The layout of the top view diagram 630 describes how objects at varying distances from the imaging device may be perceived differently due to the projective geometry of the camera.

[0077] When comparing the top view diagram 630 to the view from camera diagram 640, even if objects in the real world undergo the same displacement, their perceived motion may differ. In the view from camera diagram 640, the foreground object 510F, being closer to the camera, may exhibit a much larger apparent displacement than the background object 610B. This effect may be a consequence of perspective projection, where closer objects appear to move more dramatically than those farther away.

[0078] In an embodiment, the measurement module 420 may be configured to analyze differences in motion fields across successive frames (520N, 520N-1, 520N-2) and segment the foreground object 510F from the background. By tracking the pixel displacement associated with moving objects, the measurement module 420 may generate the velocity map 510, which serves as an input for motion segmentation and object tracking. This capability may enable accurate identification of dynamic elements within the captured scene 100S, supporting various applications such as depth estimation, motion-based object detection, and scene understanding. Figure 6C illustrates an optical flow map 620, in accordance with an exemplary embodiment of the present disclosure. In an embodiment, the measurement module 420 is configured for obtaining (e.g. generating, creating) the optical flow-map 620 for each of the identified foreground objects 510F. The optical flow-map 620 comprises displacement values of the moving pixels in the obtained (e.g. captured) frames 520N, 520N-1, 520N-2). The optical flow-map 620 indicates the amount of movement of each pixel in the obtained (e.g. captured) frames (520N, 520N-1, 520N-2). In an embodiment, the measurement module 420 is configured for obtaining (e.g. generating, creating) the velocity map 510 based on the optical flow-map 620 and a time duration elapsed between the obtained frames 520N-1 and 520N-2.

[0079] Figures 7A-7B illustrate the working of the buffering rate module 430 with reference to the velocity map 510, in accordance with an embodiment of the present disclosure. In an embodiment, the buffering rate module 430 is configured for determining a movement score by correlating a count of pixels for which the velocity value is greater than a predetermined (e.g. defined) threshold velocity value and a total number of the pixels in the velocity map 510.

[0080] Figure 7A illustrates an exemplary set-up for determination of the predetermined (e.g. defined) threshold velocity value, in accordance with an embodiment of the present disclosure. It is known that a human eye is naturally not capable of detecting movement of less than 0.5 degree / s. Based on this natural phenomenon, a minimum linear velocity for the human eye to depict or perceive any movement may be determined (e.g. calculated) using the following equations (1) and (2):

[0081] Minimum linear velocity = d * w ... (1)

[0082] Minimum linear velocity (in pixel) = Pixel density * Linear velocity = R / L * d* w ... (2)

[0083] wherein:

[0084] d = Distance between the eye and the screen (Optical Path).

[0085] w = Minimum recognizable angular speed (Angle from eye axis)

[0086] L = Length and Width of the Screen (L x L)

[0087] R = Resolution of screen (R x R)

[0088] Pixel Density = R / L

[0089] The minimum linear velocity determined (e.g. obtained, calculated) above may be chosen as the predetermined (e.g. defined) threshold velocity value. In an embodiment, the buffering rate module 430 is configured for determining the movement score by correlating the count of pixels for which the velocity value is greater than the minimum linear velocity calculated above and a total number of the pixels in the velocity map 510.

[0090] Referring again to Figure 7B, the buffering rate module 430 is configured for counting the pixels for which the velocity value is greater than the predetermined (e.g. defined) threshold velocity value. The buffering rate module 430 is further configured for determining (e.g. calculating) a ratio of the counted pixels and the total number of the pixels in the velocity map 510 as the movement score 710. The movement score 710 represents (e.g. indicates) the noticeable movement for the human eye in terms of moving pixels. The movement score 710 may be reported as a percentage of the obtained (e.g. captured) frames (520N, 520N-1, 520N-2), wherein, the movement score 710 may indicate the percentage of the frame that is considered moving for the device 150 (or, the system which is communicatively coupled with the device 150). The movement score 710 may be between 0 to 100. Based on the movement score, the partial frame fill percentage 720 of the current frame may be determined. The partial frame fill percentage 720 may be determined as maximum of 20 and movement score.

[0091] In an embodiment, the movement score 710 is determined (e.g. calculated) for every frame obtained (e.g. captured). In an embodiment, the movement score is determined (e.g. calculated) for every frame where movement is identified (e.g. detected) by the detection module 410. In an embodiment, the movement score 710 is determined (e.g. calculated) for every second or third frame obtained (e.g. captured) based on the movement being identified (e.g. detected) by the detection module 410. In an embodiment, the movement score 710 is obtained (e.g. determined, identified, calculated) for every second or third frame obtained (e.g. captured) irrespective of the movement being detected by the detection module 410. In an embodiment, the buffering rate module 430 is further configured for increasing the buffering rate corresponding to (e.g. in proportion to) an increase in the movement score. In an embodiment, the buffering rate module 430 is further configured for decreasing the buffering rate corresponding to (e.g. in proportion to) a decrease in the movement score. The buffering rate, therefore, does not remain static and is being modified (e.g. varied) by the device 150 (or, the system which is communicatively coupled with the device 150) depending upon the scene 100S; and thus, achieving optimization of the computational resources of the device 150 including the GPU / CPU and the like.

[0092] Figure 8A illustrates the device 150 for dynamically optimizing latency for rendering the real-world scene 100S in the device 150, in an embodiment of the present disclosure. The device 150 (or, the system which is communicatively coupled with the device 150) includes the plurality of modules 400. Herein, the plurality of modules includes a renderer 840, a mode detection module 850, the measurement module 420, and the buffering rate module 430.

[0093] The measurement module 820 is configured for obtaining (e.g. generating) a pixel-velocity map such as the velocity map 510 based on pixel movement, across the first image frame 520N-2 and the second image frame 520N-1, of an object in the real-world scene 100S being obtained (e.g. captured) by the detection module 810. Upon generation of the pixel-velocity map, the measurement module 420 is further configured for determining (e.g. computing) a movement score by correlating the number of pixels for which the velocity value is higher than the predetermined (e.g. defined) threshold velocity value. The buffering rate module 430 is configured for determining (e.g. varying, modifying) the buffering rate corresponding to (e.g. in proportion to) the movement score. In an embodiment, the working of the detection module 810 may be similar to the detection module 410; the working of the measurement module 820 may be similar to the measurement module 420; and the working of the buffering rate module 830 may be similar to the buffering rate module 430.

[0094] In an embodiment, the device 150 (or, the system which is communicatively coupled with the device 150) further includes the mode detection module 850. The mode detection module 850 is configured for detecting whether a pass-through mode of the device 150 is enabled. The pass-through mode of the device 150 may be enabled in various scenarios such as a Mixed Reality scenario. In the mixed reality scenario, the attention of the user is more focused on the virtual content. The pass-through mode of the device 150 may be enabled during full an Augmented Reality (AR) scenario, wherein the user has his full attention on the AR content.

[0095] In any scenario, it is desired that the latency should be minimal. Further, the complexity of the scene 100S may also vary with respect to the scenario of the pass-through mode of the device 150. Further, the complexity of the scene 100S may be determined with respect to the scenario of the pass-through mode of the device 150. The latency requirement / tolerance may vary with the complexity of the scene 100S. Accordingly, the device 150 (or, the system which is communicatively coupled with the device 150) dynamically varies the latency for optimal usage of resources such as the CPU, GPU and the like. For example, a very low latency for mixed reality scenarios is not essential and the resources of the GPU may be freed up for other useful tasks. Similarly, when the movement score reduces and may be nearly zero as for example when the scene 100S is static, the buffering rate may also be reduced. As a result, the load on the computational resources of the device 150 is advantageously reduced. In an embodiment, the working of the renderer 840 may be explained in accordance to Figure 9A-9B.

[0096] Figure 8B illustrates a comparison of the utilization of the CPU / GPU while dynamically optimizing latency for rendering the real-world scene 100S in the device 150, in accordance with an embodiment of the present disclosure. Graph 890-1, of the related art, shows that the utilization of resources of the CPU / GPU is same / constant throughout the rendering of the scene 100S based on the static partial frame rendering policy of the related art. This leads to sub-optimal latency savings. On the other hand, graph 890-2 of the CPU / GPU resource utilization versus time shows that the utilization of the resources varies as per the changing complexity of the scene 100S being rendered. When the scene 100S is stable and less dynamic, the complexity is less. The movement score measured by the device 150 (or, the system which is communicatively coupled with the device 150) is less, based on which the buffering rate / fill percentage may be reduced. This saves on the resources of the CPU / GPU.

[0097] Figures 9A-9B illustrate the working of the renderer 840 of the device 150 (or, the system which is communicatively coupled with the device 150), in accordance with an embodiment of the present disclosure. The renderer 840 may be configured for rendering, at a buffering rate, in the device 150, a first image frame such as the frame 520N-2 and a second image frame such as the frame 520N-1 of the real-world scene 100S. In an embodiment, the buffering rate module 430 is configured for determining (e.g. modifying) the buffering rate associated with a default partial-frame rendering delivery-policy of the device 150. Herein, the buffering rate associated with the device 150 is not fixed and may be modified against the default partial-frame rendering delivery-policy of the device 150. More precisely, the system 310 dynamically modifies the buffering rate depending upon the complexity (indicated by the movement score) of the scene 100S.

[0098] Figure 9A, in particular, illustrates an exemplary partial-frame rendering delivery-policy of the device 150, which may be the default partial-frame rendering delivery-policy of the device 150. In an exemplary embodiment, the default partial-frame rendering delivery-policy of the device 150 may have a partial frame fill percentage of 25%. Accordingly, for each frame, four updates are required to be made to a buffer of the camera of the device 150. During each update, 25% of the frame is filled with new data. The individual updates of 25% each are then sent to the renderer 840 for rendering on a part of the screen 352 of the device 150.

[0099] Figure 9B illustrates exemplary warps after each of the updates as per the exemplary partial frame rendering delivery policy of Figure 9A. A first warp corresponding to a first update, a second warp corresponding to a second update and a third warp corresponding to a third update are shown. As the first wrap to fourth wrap goes on, camera buffer is filled with new values instead of old values.

[0100] Figure 10 illustrates a process flow 1000 for modifying the default partial-frame rendering delivery-policy of the device 150, in accordance with an embodiment of the present disclosure. At 902, for the frame 520N-1, a change in the complexity of scene 100S is detected. At 904, the movement score is calculated in accordance to the detected change. At 906, the default partial-frame rendering delivery-policy is determined (e.g. modified, updated) based upon the movement score. The determined partial-frame rendering delivery-policy is used for the frame 520N and so on.

[0101] Figure 11 is a flowchart illustrating a method 1100 for managing latency in a partial-frame-rendering Video-See-Through (VST) device such as the device 150, in accordance with an embodiment of the present disclosure.

[0102] Referring to Figures 3-10 together, the method 1100 may be performed by the device 150 such as a camera device having the pass-through mode, e.g., a camcorder, a mobile device, a tab with similar capabilities, and the like, based on instructions retrieved from non-transitory computer-readable media. A computer-readable media may include machine-executable or computer-executable instructions to perform all or portions of the described method. The computer-readable media may be, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable data storage media.

[0103] The method 1100 includes a series of operations shown at operation 1102 through operation 1106 of Figure 11. The method 1100 may be performed by the device 150 in conjunction with one or more modules 400, the details of which are explained in conjunction with Figures 3-10, and the same are not repeated here for the sake of brevity. The method 1100 begins at operation 1102.

[0104] At operation 1102, the method 1100 includes identifying (e.g. detecting) for moving objects in the real-world scene 100S being captured by the device 150. At operation 1104, the method 1100 includes obtaining (e.g. determining, measuring), continuously, the movement of one or more moving objects of the scene 100S. At operation 1106, the method 1100 includes determining (e.g. modifying, varying), dynamically, a buffering rate for rendering obtained (e.g. captured) frames of the scene 100S corresponding to (e.g. in proportion to) the measured movement of the one or more moving objects in the obtained (e.g. captured) frames of the scene 100S.

[0105] Hence, the method 1100 may be directed at optimizing the latency in rendering during the pass-through mode of the device 150 by dynamically changing the buffer rate depending on the complexity of the scene 100S to save CPU / GPU resources.

[0106] The method 1100 further includes obtaining (e.g. generating) the velocity map 510 indicating the complexity of the scene 100S. The velocity map 510 comprises velocity values of moving pixels associated with the one or more moving objects in the obtained (e.g. captured) frames (520N, 520N-1, 520N-2). Accordingly, the method 1100 further includes identifying at least one foreground object (such as the object 510F) from among the one or more objects (510F, 610B) by performing a segmentation of the obtained (e.g. captured) frames. The segmentation, at operation 1102, may be performed by known methods. Image segmentation may be described as a　computer vision technique　that separates a digital image into discrete groups of pixels―image segments.　Upon identifying of the foreground object 510F, the method 1100 generates the velocity map 510 associated with the foreground object 510F.

[0107] In an embodiment, the method 1100 further includes obtaining (e.g. generating, creating) the optical flow-map (such as maps 520, 620) for each of the identified foreground objects 510F. The optical flow-map 520, 620 comprises displacement values of the moving pixels of the identified foreground objects 510F in the obtained (e.g. captured) frames (520N, 520N-1, 520N-2). The velocity map 510 is obtained (e.g. generated) by correlating the displacement in the optical flow-map 520, 620 and a time duration elapsed between the obtained (e.g. captured) frames (520N, 520N-1, 520N-2).

[0108] At operation 1106, the method 1100 further includes determining a movement score by correlating a count of pixels for which the velocity value is greater than a predetermined (e.g. defined) threshold velocity value and a total number of the pixels in the velocity map 510. Upon determination of the movement score, the method 1100 further includes determining (e.g. varying), the buffering rate, corresponding to (e.g. in proportion to) the movement score.

[0109] In an embodiment, the method 1100 further includes continuously determining the movement score for the obtained (e.g. captured) frames (520N, 520N-1, 520N-2). For the obtained (e.g. captured) frame 520N, previous frames, the frame 520N-1 and 520N-2 are used for determining (e.g. computing, calculating) the movement score. Similarly, for the obtained (e.g. captured) frame 520N-1, previous frames, the frame 520N-2 and a frame 520N-3 are used for determining (e.g. computing, calculating) the movement score.

[0110] Based upon the movement score, the method 110 further includes performing one of: increasing the buffering rate corresponding to (e.g. in proportion to) an increase in the movement score; and decreasing the buffering rate corresponding to (e.g. in proportion to a decrease in the movement score.

[0111] Figure 12 is a flowchart illustrating a method 1200 for dynamically optimizing latency for rendering the scene 100S in the device 150 in a partial-frame-rendering Video-See-Through (VST) device such as the device 150, in accordance with an embodiment of the present disclosure. In an embodiment, the method 1200 may be performed by the device 150 after the method 1100. In an embodiment, the method 1200 may be performed by the device 150 independently of the method 1100.

[0112] Referring to Figures 3-11 together, the method 1200 may be performed by the device 150 such as a camera device having the pass-through mode such as a camcorder, a mobile device, a tab with similar capabilities, and the like, based on instructions retrieved from computer-readable media. A computer-readable media may include machine-executable or computer-executable instructions to perform all or portions of the described method. The computer-readable media may be, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable data storage media. The computer-readable media may be non-transitory.

[0113] A "non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

[0114] The method 1200 includes a series of operations shown at operation 1202 through operation 1208 of Figure 12. The method 1200 may be performed by the device 150 or, the system which is communicatively coupled with the device 150, the details of which are explained in conjunction with Figures 3-10, and the same are not repeated here for the sake of brevity. The method 1200 begins at operation 1202.

[0115] At operation 1202, the method 1200 includes rendering, at a buffering rate, in the VST device, a first image frame such as the frame 520N-1 and a second image frame such as the frame 520N-2 of the scene 100S being obtained (for example, captured) by the device 150. At operation 1204, the method 1200 further includes obtaining (for example, generating) a pixel-velocity map such as the velocity map 510 based on pixel movement, across the first image frame 520N-1and the second image frame 520N-2, of an object (such as the objects 510F, 610B) in the scene 100S. At operation 1206, the method 1200 further includes obtaining (e.g. determining, computing, identifying) a movement score by correlating a number of pixels for which the velocity value is higher than a predetermined (e.g. defined) threshold velocity value in the pixel-velocity map. Based upon the obtained movement score, the method 1200 further includes determining (e.g. modifying) the buffering rate corresponding to (e.g. in proportion to) the movement score.

[0116] In an embodiment, the method 1200 may further include detecting whether the pass-through mode of the device 150 is enabled.

[0117] In an embodiment, the method 1200 may further include selecting at least one foreground object such as the object 510F from the objects (510F, 610B) in the scene 100S. Herein, the method 1200 may further include obtaining (e.g. generating) the velocity map 510 by determining (e.g. computing, calculating) velocity values for each pixel of the at least one foreground object 510F. The method 1200 further includes determining (e.g. calculating) the velocity values by correlating a displacement of the pixel movement with the time elapse between the first image frame 520N-2 and the second image frame 520N-1.

[0118] In an embodiment, the method 1200 may further include determining (e.g. modifying) the buffering rate by modifying a default partial-frame rendering delivery-policy of the device 150. The device 150 may have the default partial-frame rendering delivery-policy such as one explained with reference to Figure 9.

[0119] The device, system and method of the present disclosure may take advantage of the measured change in the complexity of the scene 100S and based on the scene dynamism measured as the movement score, change the buffering rate for the partial frame fill policy. More precisely, the device, system and method of the present disclosure may dynamically modify the amount of buffering of frames to be rendered in the device 150 such as the HMD based on the movements of objects in the scene 100S in real-time. As a result, the device, system and method of the present disclosure are able to reduce the latency and at the same time optimize the load on the computational resources of the device 150.

[0120] The static partial frame delivery mechanism has the same amount of CPU and GPU utilization irrespective of the scene complexity. The system and method of the invention changes the buffering rate dynamically so that the GPU and CPU save their computations when the scene is not changing much. As a result, the invention achieves optimized rendering and GPU utilization in the device. The invention aims at saving the GPU resources when it is not necessary to employ them. In the traditional partial frame delivery, the resource utilization remains static for all scenes, even when not needed to be at that level. Consequently, the power consumption in the device 150 is also optimized. Further, the device 150 is able to run multiple applications smoothly in the pass-through mode as the memory is also available and free.

[0121] According to an embodiment of the disclosure, a method may be disclosed. In an embodiment, the method may include identifying one or more moving objects in a real-world scene being obtained. The method may include obtaining, continuously, the movement of the one or more moving objects. The method may include determining, a buffering rate for rendering obtained frames of the real-world scene, corresponding to the obtained movement of the one or more moving objects.

[0122] According to an embodiment of the disclosure, the method may include obtaining a velocity map. The velocity map may comprise velocity values of moving pixels corresponding to the one or more moving objects in the obtained frames.

[0123] According to an embodiment of the disclosure, the method may include identifying at least one foreground object from among the one or more objects by performing a segmentation of the obtained frames. According to an embodiment of the disclosure, the method may include obtaining the velocity map for the identified foreground object.

[0124] According to an embodiment of the disclosure, the method may include obtaining an optical flow-map for each of the identified foreground objects. The optical flow-map may include displacement values of the moving pixels in the obtained frames. The method may include obtaining the velocity map based on the optical flow-map and a time duration elapsed between the obtained frames.

[0125] According to an embodiment of the disclosure, the method may include determining a movement score by correlating a count of pixels for which the velocity value is greater than a determined threshold velocity value and a total number of the pixels in the velocity map. The method may include determining, the buffering rate, corresponding to the movement score.

[0126] According to an embodiment of the disclosure, the method may include determining the movement score for the obtained frames. The method may include increasing the buffering rate corresponding to an increase in the movement score. The method may include performing decreasing the buffering rate corresponding to a decrease in the movement score.

[0127] According to an embodiment of the disclosure, the method may include rendering, at the buffering rate, in the VST device, a first image frame and a second image frame of the real-world scene being obtained. The method may include obtaining a pixel-velocity map based on pixel movement, across a first image frame and a second image frame, of an object in the real-world scene being obtained. According to an embodiment of the disclosure, the method may include identifying a movement score by correlating a number of pixels for which the velocity value is higher than a defined threshold velocity value in the pixel-velocity map. According to an embodiment of the disclosure, the method may include determining the buffering rate corresponding to the movement score.

[0128] According to an embodiment of the disclosure, a VST device may be disclosed. In an embodiment, the VST device may include a camera. The VST device may include at least one processor comprising processing circuitry. The VST device may include at least one memory including one or more instructions. In an embodiment, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to identify one or more moving objects in a real-world scene being obtained. In an embodiment, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to obtain, the movement of the one or more moving objects. In an embodiment, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to determine, a buffering rate for rendering obtained frames of the real-world scene, corresponding to the obtained movement of the one or more moving objects

[0129] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to obtain a velocity map, wherein the velocity map comprises velocity values of moving pixels corresponding to the one or more moving objects in the obtained frames.

[0130] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to identify at least one foreground object from among the one or more objects by performing a segmentation of the obtained frames. According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to obtain the velocity map for the identified foreground object.

[0131] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to obtain an optical flow-map for each of the identified foreground objects, wherein the optical flow-map comprises displacement values of the moving pixels in the obtained frames. According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to obtain the velocity map based on the optical flow-map and a time duration elapsed between the captured frames.

[0132] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to determine a movement score by correlating a count of pixels for which the velocity value is greater than a determined threshold velocity value and a total number of the pixels in the velocity map. According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to determine, the buffering rate, corresponding to the movement score.

[0133] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to increase the buffering rate corresponding to an increase in the movement score. According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to decrease the buffering rate corresponding to a decrease in the movement score.

[0134] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to render, at the buffering rate, in the VST device, a first image frame and a second image frame of the real-world scene being obtained. According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to obtain a pixel-velocity map based on pixel movement, across a first image frame and a second image frame, of an object in the real-world scene being obtained. According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to identify a movement score by correlating a number of pixels for which the velocity value is higher than a defined threshold velocity value in the pixel-velocity map. According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device to determine the buffering rate corresponding to the movement score.

[0135] According to an embodiment of the disclosure, a computer-readable medium containing instructions is disclosed. The instructions, when executed by at least one processor, cause the VST device to perform the method provided.

[0136] According to an embodiment of the disclosure, a method for managing latency in a partial-frame-rendering Visual-See-Through (VST) device is provided. In an embodiment, the method may include detecting for moving objects in a real-world scene being captured. In an embodiment, the method may include measuring, continuously, the movement of one or more moving objects. In an embodiment, the method may include varying, dynamically, a buffering rate for rendering captured frames of the real-world scene. In an embodiment, the buffering rate may be varied in proportion to the measured movement of the one or more moving objects.

[0137] According to an embodiment of the disclosure, the method may include generating a velocity map. The velocity map may include velocity values of moving pixels associated with the one or more moving objects in the captured frames.

[0138] According to an embodiment of the disclosure, the method may include identifying at least one foreground object from among the one or more objects by performing a segmentation of the captured frames. In an embodiment, the method may include generating the velocity map for the identified foreground object.

[0139] According to an embodiment of the disclosure, the method may include creating an optical flow-map for each of the identified foreground objects, the optical flow-map comprising displacement values of the moving pixels in the captured frames. In an embodiment, the method may include creating the velocity map based on the optical flow-map and a time duration elapsed between the captured frames.

[0140] According to an embodiment of the disclosure, the method may include determining a movement score by correlating a count of pixels for which the velocity value is greater than a predetermined threshold velocity value and a total number of the pixels in the velocity map. According to an embodiment of the disclosure, the method may include varying, the buffering rate, in proportion to the movement score.

[0141] According to an embodiment of the disclosure, the method may include determining, continuously, the movement score for the captured frames. In an embodiment, the method may include increasing the buffering rate in proportion to an increase in the movement score. In an embodiment, the method may include decreasing the buffering rate in proportion to a decrease in the movement score.

[0142] According to an embodiment of the disclosure, a method for dynamically optimizing latency for rendering a real-world scene in a Visual See Through (VST) device is provided. In an embodiment, the method may include rendering, at a buffering rate, in the VST device, a first image frame and a second image frame of the real-world scene being captured. In an embodiment, the method may include generating a pixel-velocity map based on pixel movement, across the first image frame and the second image frame, of an object in the real-world scene being captured. In an embodiment, the method may include generating a pixel-velocity map based on pixel movement, across the first image frame and the second image frame, of an object in the real-world scene being captured. In an embodiment, the method may include computing a movement score by correlating a number of pixels for which the velocity value is higher than a predetermined threshold velocity value in the pixel-velocity map. In an embodiment, the method may include modifying the buffering rate in proportion to the movement score.

[0143] According to an embodiment of the disclosure, the method may include detecting whether a pass-through mode of the VST device is enabled. In an embodiment, the method may include selecting at least one foreground object from the objects in the real-world scene. In an embodiment, the method may include calculating velocity values for each pixel of the at least one foreground object. In an embodiment, the method may include correlating a displacement of the pixel movement with the time elapse between the first image frame and the second image frame. According to an embodiment of the disclosure, the method may include modifying a default partial-frame rendering delivery-policy of the VST device.

[0144] According to an embodiment of the disclosure, a system for managing latency in a partial-frame-rendering Visual-See-Through (VST) device is provided. In an embodiment, the system may include a detection module (410) configured for detecting for moving objects in a real-world scene being captured. The system may include a measurement module configured for measuring, continuously, the movement of one or more moving objects. The system may include a buffering rate module configured for varying, dynamically, a buffering rate for rendering captured frames of the real-world scene. The buffering rate may be varied in proportion to the measured movement of the one or more moving objects.

[0145] According to an embodiment of the disclosure, the system may include the measurement module configured for generating a velocity map. The velocity map may comprise velocity values of moving pixel associated with the one or more moving objects in the captured frames.

[0146] According to an embodiment of the disclosure, the system may include the measurement module configured for identifying at least one foreground object from among the one or more objects by performing a segmentation of the captured frames. According to an embodiment of the disclosure, the system may include the measurement module configured for generating the velocity map (510) for the identified foreground object.

[0147] According to an embodiment of the disclosure, the system may include the measurement module configured for creating an optical flow-map for each of the identified foreground objects, the optical flow-map comprising displacement values of the moving pixels in the captured frames. According to an embodiment of the disclosure, the system may include the measurement module configured for creating the velocity map based on the optical flow-map and a time duration elapsed between the captured frames.

[0148] According to an embodiment of the disclosure, the system may include the buffering rate module configured for determining a movement score by correlating a count of pixels for which the velocity value is greater than a predetermined threshold velocity value and a total number of the pixels in the velocity map. According to an embodiment of the disclosure, the system may include the buffering rate module configured for varying, the buffering rate, in proportion to the movement score.

[0149] According to an embodiment of the disclosure, the system may include the buffering rate module configured for determining, continuously, the movement score for the captured frames. According to an embodiment of the disclosure, the system may include the buffering rate module configured for performing increasing the buffering rate in proportion to an increase in the movement score. According to an embodiment of the disclosure, the system may include the buffering rate module configured for decreasing the buffering rate in proportion to a decrease in the movement score.

[0150] According to an embodiment of the disclosure, a system for dynamically optimizing latency for rendering a real-world scene in a Visual See Through (VST) device is provided. In an embodiment, the system may include a renderer configured for rendering, at a buffering rate, in the VST device, a first image frame and a second image frame of the real-world scene being captured. In an embodiment, the system may include a measurement module configured for generating a pixel-velocity map based on pixel movement, across the first image frame and the second image frame, of an object in the real-world scene being captured. In an embodiment, the system may include a measurement module configured for computing a movement score by correlating a number of pixels for which the velocity value is higher than a predetermined threshold velocity value in the pixel-velocity map. According to an embodiment of the disclosure, the system may include a buffering rate module configured for modifying the buffering rate in proportion to the movement score.

[0151] According to an embodiment of the disclosure, the system may include a mode detection module configured for detecting whether a pass-through mode of the VST device is enabled. According to an embodiment of the disclosure, the measurement module may be configured for selecting at least one foreground object from the objects in the real-world scene. According to an embodiment of the disclosure, the measurement module may be configured for calculating velocity values for each pixel of the at least one foreground object.

[0152] According to an embodiment of the disclosure, the measurement module may be configured for correlating a displacement of the pixel movement with the time elapse between the first image frame and the second image frame.

[0153] According to an embodiment of the disclosure, the buffering rate module may be configured for modifying a default partial-frame rendering delivery-policy of the VST device.

[0154] While specific language has been used to describe the disclosure, any limitations arising on account of the same are not intended. As would be apparent to a person in the art, various working modifications may be made to the method in order to implement the inventive concept as taught herein.

[0155] The drawings and the forgoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, orders of processes described herein may be changed and are not limited to the manner described herein.

[0156] Moreover, the actions of any flow diagram need not be implemented in the order shown; nor do all of the acts necessarily need to be performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of embodiments is by no means limited by these specific examples. Numerous variations, whether explicitly given in the specification or not, such as differences in structure, dimension, and use of material, are possible. The scope of embodiments is at least as broad as given by the following claims.

[0157] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any component(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component of any or all the claims.

Claims

1.A method (1100) comprising:identifying (1102) one or more moving objects in a real-world scene (100S) being obtained;obtaining (1104), the movement of the one or more moving objects; anddetermining(1106), a buffering rate for rendering obtained frames (520N, 520N-1, 520N-2) of the real-world scene (100S), corresponding to the obtained movement of the one or more moving objects,2.The method (1100) as claimed in claim 1, wherein obtaining the movement of the one or more moving objects comprises obtaining a velocity map (510); and wherein the velocity map (510) comprises velocity values of moving pixels corresponding to the one or more moving objects in the obtained frames (520N, 520N-1, 520N-2).3.The method (1100) as claimed in claim 2, wherein obtaining the velocity map (510) comprises:identifying at least one foreground object (510F) from among the one or more objects by performing a segmentation of the obtained frames (520N, 520N-1, 520N-2); andobtaining the velocity map (510) for the identified foreground object (510F).4.The method (1100) as claimed in any one of claims 2 to 3, wherein obtaining the velocity map (510) comprises:obtaining an optical flow-map for each of the at least one identified foreground object (510F), wherein the optical flow-map comprises displacement values of the moving pixels in the obtained frames (520N, 520N-1, 520N-2); andobtaining the velocity map (510) based on the optical flow-map and a time duration elapsed between the obtained frames (520N, 520N-1, 520N-2).5.The method (1100) as claimed in any one of claims 1 to 4, wherein determining the buffering rate comprises:determining a movement score by correlating a count of pixels for which the velocity value is greater than a defined threshold velocity value and a total number of the pixels in the velocity map (510); anddetermining, the buffering rate, corresponding to the movement score.6.The method (1100) as claimed in any one of claims 1 to 5, wherein determining the buffering rate comprises:determining the movement score for the obtained frames (520N, 520N-1, 520N-2) and performing at least one of:increasing the buffering rate corresponding to an increase in the movement score; anddecreasing the buffering rate corresponding to a decrease in the movement score.7.The method (1100) as claimed in any one of claims 1 to 6, further comprising:rendering, at the buffering rate, in the VST device (150), a first image frame and a second image frame of the real-world scene (100S) being obtained; andobtaining a pixel-velocity map (510) based on pixel movement, across a first image frame and a second image frame, of an object in the real-world scene (100S) being obtained;identifying a movement score by correlating a number of pixels for which the velocity value is higher than a defined threshold velocity value in the pixel-velocity map (510); anddetermining the buffering rate corresponding to the movement score.8.A Video-See-Through (VST) device (150) comprising:a camera (802);at least one processor (804) comprising processing circuitry;at least one memory (808) including one or more instructions, executed by the at least one processor individually or collectively, to cause the VST device (150) to:identify one or more moving objects in a real-world scene (100S) being obtained;obtain the movement of the one or more moving objects; anddetermine, a buffering rate for rendering obtained frames (520N, 520N-1, 520N-2) of the real-world scene (100S), corresponding to the obtained movement of the one or more moving objects.9.The VST device (150) as claimed in claim 8, wherein the one or more instructions, executed by the at least one processor (804) individually or collectively, to cause the VST device (150) to:obtain a velocity map (510), wherein the velocity map (510) comprises velocity values of moving pixels corresponding to the one or more moving objects in the obtained frames (520N, 520N-1, 520N-2).10.The VST device (150) as claimed in any one of claims 8 to 9, wherein the one or more instructions, executed by the at least one processor (804) individually or collectively, to cause the VST device (150) to:identify at least one foreground object from (510F) among the one or more objects by performing a segmentation of the obtained frames (520N, 520N-1, 520N-2); andobtain the velocity map (510) for the identified foreground object (510F).11.The VST device (150) as claimed in any one of claims 9 to 10, wherein the one or more instructions, executed by the at least one processor (804) individually or collectively, to cause the VST device (150) to:obtain an optical flow-map for each of the at least one identified foreground object (510F), wherein the optical flow-map comprises displacement values of the moving pixels in the obtained frames (520N, 520N-1, 520N-2); andobtain the velocity map (510) based on the optical flow-map and a time duration elapsed between the obtained frames (520N, 520N-1, 520N-2).12.The VST device (150) as claimed in any one of claims 8 to 11, wherein the one or more instructions, executed by the at least one processor (804) individually or collectively, to cause the VST device (150) to:determine a movement score by correlating a count of pixels for which the velocity value is greater than a defined threshold velocity value and a total number of the pixels in the velocity map (510); anddetermine, the buffering rate, corresponding to the movement score.13.The VST device (150) as claimed in any one of claims 8 to 12, wherein the one or more instructions, executed by the at least one processor (804) individually or collectively, to cause the VST device (150) to:determine the movement score for the obtained frames (520N, 520N-1, 520N-2); and performing at least one of:increasing the buffering rate corresponding to an increase in the movement score; anddecreasing the buffering rate corresponding to a decrease in the movement score.14.The VST device (150) as claimed in any one of claims 8 to 13, wherein the one or more instructions, executed by the at least one processor (804) individually or collectively, to cause the VST device (150) to:render, at the buffering rate, a first image frame and a second image frame of the real-world scene (100S) being obtained;obtain a pixel-velocity map (510) based on pixel movement, across a first image frame and a second image frame, of an object in the real-world scene (100S) being obtained;identify a movement score by correlating a number of pixels for which the velocity value is higher than a defined threshold velocity value in the pixel-velocity map (510); anddetermine the buffering rate corresponding to the movement score.15.A computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the VST device (150) to perform the method of any one of claims 1 to 7.

Images