System and method for streaming-based object recognition

Abstract

Systems and methods for streaming-based object recognition may include a device that transmits video data of the device at a first instance of time and first coordinates of a first viewport to a server. The video data may include one or more objects in the first viewport. The device may transmit a second coordinate of a second viewport of the device at a second time instance. The device may receive data from the server corresponding to one or more objects within the second viewport. The data may be received in order according to coordinates of the one or more objects relative to the second viewport. The device may render data related to the one or more objects within a third viewport at a third time instance.

Description

Technical Field

[0001] The present disclosure generally relates to augmented reality (AR) devices and virtual reality (VR) devices, including but not limited to streaming-based object recognition for AR / VR devices. Background Art

[0002] Some devices may have augmented reality (AR) functionality or virtual reality (VR) functionality. For example, some devices may provide functionality that allows a user or wearer of the device to interact in a virtual reality or augmented reality environment based on objects located and detected in the physical environment of the device. Summary of the invention

[0003] According to a first aspect of the present disclosure, a method is provided, which includes: a device transmitting video data and first coordinates of a first viewport of the device at a first time instance to a server, the video data including one or more objects in the first viewport; the device transmitting second coordinates of a second viewport of the device at a second time instance; the device receiving data corresponding to one or more objects in the second viewport from the server, the data being received in sequence according to the coordinates of the one or more objects relative to the second viewport; and the device rendering data related to the one or more objects in a third viewport at a third time instance.

[0004] In some embodiments, the method may further include: receiving, by the device, data from the server in sequence according to the coordinates of the one or more objects relative to the reference point in the second viewport. In some embodiments, the one or more objects may include a first object located at a first distance from the reference point and a second object located at a second distance from the reference point. The method may further include: when the second distance is greater than the first distance, receiving, by the device, data of the first object before receiving data of the second object from the server. In some embodiments, the method may further include: determining, by the device, a first coordinate of the first viewport based on the orientation of the device using sensor data of the device.

[0005] In some embodiments, the method may further include: establishing, by the device, one or more parameters for processing or transmitting at least one of the video data with the server, and providing the video data according to the one or more established parameters. In some embodiments, the method may further include: determining, by the device, whether the available computing resources of the device meet the threshold criteria. In some embodiments, the method may further include: at a fourth time instance, in response to the available computing resources meeting the threshold criteria: detecting an object from the second video data by the device; determining, by the device, one or more features of the object; and transmitting, by the device, the coordinates of the object and the one or more features to the server. In some embodiments, the one or more objects may include at least one object located within the first viewport and outside the second viewport. The method may further include: determining, by the device, that the at least one object is located within a third viewport, and rendering, by the device, data related to the at least one object within the third viewport. In some embodiments, the one or more objects include at least one object located within the second viewport. The method may further include: determining, by the device, that the at least one object is located outside the third viewport, and abandoning, by the device, rendering data corresponding to the at least one object within the third viewport.

[0006] According to a second aspect of the present disclosure, a device is provided, the device comprising a transceiver and at least one processor. The transceiver is configured to: transmit video data of the device at a first time instance and a first coordinate of a first viewport to a server, the video data comprising one or more objects in the first viewport; transmit the second coordinate of a second viewport of the device at a second time instance to the server; receive data corresponding to one or more objects in the second viewport from the server, the data being received in sequence according to the coordinates of the one or more objects relative to the second viewport. The at least one processor is configured to render data related to the one or more objects in a third viewport at a third time instance.

[0007] In some embodiments, the transceiver may be configured to sequentially receive data from the server based on the coordinates of the one or more objects relative to the reference point in the second viewport. In some embodiments, the one or more objects may include a first object located at a first distance from the reference point and a second object located at a second distance from the reference point. The transceiver may also be configured to receive data of the first object before receiving data of the second object from the server when the second distance is greater than the first distance.

[0008] In some embodiments, the at least one processor may be configured to use sensor data of the device to determine the first coordinates of the first viewport based on the orientation of the device. In some embodiments, the at least one processor may be configured to establish one or more parameters for processing or transmitting at least one of the video data with the server, and provide the video data according to the one or more established parameters. In some embodiments, the at least one processor may be configured to determine whether the available computing resources of the device meet the threshold criteria at a fourth time instance. In some embodiments, the at least one processor may be configured to detect an object from the second video data in response to the available computing resources meeting the threshold criteria, and determine one or more features of the object. The transceiver may also be configured to transmit the coordinates and one or more features of the object to the server. In some embodiments, the one or more objects may include at least one object located within the first viewport and outside the second viewport. The at least one processor may be configured to determine that at least one object is within the third viewport, and data related to the at least one object may be rendered within the third viewport. In some embodiments, the one or more objects include at least one object located within the second viewport. The at least one processor may be configured to determine that the at least one object is outside the third viewport, and data corresponding to the at least one object may be abandoned in rendering within the third viewport.

[0009] According to a third aspect of the present disclosure, a non-transitory computer-readable medium is provided, which stores instructions that, when executed by at least one processor, cause the at least one processor to: transmit video data and first coordinates of a first viewport of the device at a first time instance to a server via a transceiver, the video data including one or more objects in the first viewport; transmit second coordinates of a second viewport of the device at a second time instance to the server via a transceiver; receive data corresponding to one or more objects in the second viewport from the server via the transceiver, the data being received in sequence according to the coordinates of the one or more objects relative to the second viewport; and render data related to the one or more objects in a third viewport at a third time instance.

[0010] In some embodiments, the instructions may further cause the one or more processors to sequentially receive data from the server based on coordinates of the one or more objects relative to the reference point in the second viewport. In some embodiments, the one or more objects may include a first object located at a first distance from the reference point and a second object located at a second distance from the reference point. When the second distance is greater than the first distance, the instructions may further cause the one or more processors to receive data for the first object before receiving data for the second object from the server.

[0011] It will be understood that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be generalizable to any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. The above general description and the following detailed description are exemplary and illustrative only, and are not limitations of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and names in the various drawings indicate like elements. For clarity, not every component is labeled in every drawing.

[0013] Figure 1 is a schematic diagram of a system environment including an artificial reality system according to one or more embodiments of the present disclosure.

[0014] Figure 2 is a schematic diagram of a head mounted display according to one or more embodiments of the present disclosure.

[0015] Figure 3 is a block diagram of a computing environment according to one or more embodiments of the present disclosure.

[0016] Figure 4 is a network diagram illustrating a network path between a device and an identification server according to one or more embodiments of the present disclosure.

[0017] Figure 5 is a graph illustrating remaining computing resources of a device relative to the number of objects in an environment of the device according to one or more embodiments of the present disclosure.

[0018] Figure 6 is a block diagram of a system for streaming-based object recognition according to one or more embodiments of the present disclosure.

[0019] Figure 7 is a flowchart illustrating a method for streaming-based object recognition according to one or more embodiments of the present disclosure.

[0020] FIG. 8A to FIG. 8D A series of views of an environment of a device according to one or more embodiments of the present disclosure are shown.

[0021] Fig. 9 is a flowchart illustrating a method for transmitting a recognition result according to one or more embodiments of the present disclosure.

[0022] Fig.10 is a flowchart illustrating a method for streaming-based object recognition according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Before turning to the drawings that show certain embodiments in detail, it should be understood that the present disclosure is not limited to the details or methods set forth in the specification or shown in the drawings. It should also be understood that the terminology used herein is for descriptive purposes only and should not be considered as limiting.

[0024] For the purpose of reading the following description of various embodiments, the following description of the various parts of the specification and their corresponding contents may be helpful:

[0025] Section A discloses an artificial reality system that can be used to practice the embodiments described herein;

[0026] Section B discloses a computing system that can be used to implement various aspects of the present disclosure; and

[0027] Section C discloses systems and methods for streaming-based object recognition.

[0028] A. Artificial Reality Systems

[0029] Disclosed herein are systems and methods for facilitating distribution of artificial reality (eg, augmented reality (AR), virtual reality (VR), or mixed reality (MR)) content. Figure 1 1 is a block diagram of an example artificial reality system environment 100. In some embodiments, the artificial reality system environment 100 includes a head wearable display (HWD) 150 worn by a user, and a console 110 that provides artificial reality content to the HWD 150. The HWD 150 can detect its position and / or orientation of the HWD 150, the position and / or orientation of the HWD 150, and user input for the artificial reality to generate image data indicating an image of the artificial reality, and send the image data to the HWD 150 for presentation.

[0030] In some embodiments, the artificial reality system environment 100 includes Figure 1 More, fewer, or more of the components shown may be included. Figure 1 In some embodiments, the functionality of one or more components in the artificial reality system environment 100 may be distributed among the components in a manner different from that described herein. For example, some of the functions of the console 110 may be performed by the HWD 150. For example, some of the functions of the HWD 150 may be performed by the console 110. In some embodiments, the console 110 is integrated as part of the HWD 150.

[0031] In some embodiments, HWD 150 is an electronic component that can be worn by a user and can present or provide an artificial reality experience to the user. HWD 150 can be referred to as, include, or be part of a head mounted display (HMD), a head mounted device (HMD), a head wearable device (HWD), a head worn display (HWD), or a head worn device (HWD). HWD 150 can render one or more images, videos, audio, or some combination thereof to provide an artificial reality experience to the user. In some embodiments, audio is presented by an external device (e.g., a speaker and / or headphones) that receives audio information from HWD 150, console 110, or both, and presents audio based on the audio information. In some embodiments, HWD 150 includes a sensor 155, an eye tracker 160, a hand tracker 162, a communication interface 165, an image renderer 170, an electronic display 175, a lens 180, and a compensator 185. These components may function together to detect the position of the HWD 150 and the gaze direction of the user wearing the HWD 150, and may render an image of a field of view corresponding to the detected position and / or orientation of the HWD 150 within the artificial reality. Figure 1 More, fewer, or more of the components shown may be included. Figure 1 The parts shown are different parts.

[0032] In some embodiments, the sensor 155 includes electronic components, or a combination of electronic components and software components, that detect the position and orientation of the HWD 150. Examples of the sensor 155 may include: one or more imaging sensors, one or more accelerometers, one or more gyroscopes, one or more magnetometers, or another suitable type of sensor that detects motion and / or position. For example, one or more accelerometers may measure translational motion (e.g., forward / backward, up / down, left / right), and one or more gyroscopes may measure rotational motion (e.g., pitch, yaw, roll). In some embodiments, the sensor 155 detects translational motion and rotational motion and determines the orientation and position of the HMD 150. In one aspect, the sensor 155 may detect translational motion and rotational motion relative to the previous orientation and position of the HMD 150, and determine the new orientation and / or position of the HMD 150 by accumulating or integrating the detected translational motion and / or rotational motion. For example, assuming that HWD 150 is facing a direction 25 degrees from a reference direction, sensor 155 may determine that HWD 150 is now facing or facing a direction 45 degrees from the reference direction in response to detecting that HWD 150 has turned 20 degrees. For another example, assuming that HWD 150 is located two feet from a reference point in a first direction, sensor 155 may determine that HWD 150 is now located at the vector product of two feet in the first direction and three feet in the second direction in response to detecting that HWD 150 has moved three feet in a second direction.

[0033] In some embodiments, each eye tracker 160 includes an electronic component, or a combination of electronic components and software components, that determines the gaze direction of the user of the HWD 150. In some embodiments, the HWD 150, the console 110, or a combination thereof can generate image data for artificial reality in conjunction with the gaze direction of the user of the HWD 150. In some embodiments, each eye tracker 160 includes two eye trackers, wherein each eye tracker 160 captures an image of a corresponding eye and determines the gaze direction of the eye. In one example, the eye tracker 160 determines the angular rotation of the eye, the translation of the eye, the torsion change of the eye, and / or the shape change of the eye based on the captured image of the eye, and determines the relative gaze direction relative to the HWD 150 based on the determined angular rotation, translation, and torsion change of the eye. In one approach, the eye tracker 160 can illuminate or project a predetermined reference pattern or structure pattern on a portion of the eye, and capture an image of the eye to analyze the pattern projected on the portion of the eye, thereby determining the relative gaze direction of the eye relative to the HWD 150. In some embodiments, eye tracker 160 determines the user's gaze direction in conjunction with the orientation of HWD 150 and the relative gaze direction relative to HWD 150. For example, assuming that HWD 150 is oriented in a direction 30 degrees from a reference direction, and the relative gaze direction of HWD 150 is -10 degrees (or 350 degrees) relative to HWD 150, eye tracker 160 may determine that the user's gaze direction is 20 degrees from the reference direction. In some embodiments, the user of HWD 150 may configure HWD 150 (e.g., through user settings) to enable or disable eye tracker 160. In some embodiments, the user of HWD 150 is prompted to enable or disable eye tracker 160.

[0034] In some embodiments, hand tracker 162 includes electronic components, or a combination of electronic components and software components, that track the user's hands. In some embodiments, hand tracker 162 includes or is coupled to an imaging sensor (e.g., a camera) and an image processor that can detect the shape, position, and orientation of the hand. Hand tracker 162 can generate hand tracking measurements that indicate the shape, position, and orientation of the detected hand.

[0035] In some embodiments, the communication interface 165 includes an electronic component, or a combination of an electronic component and a software component, that communicates with the console 110. The communication interface 165 can communicate with the communication interface 115 of the console 110 via a communication link. The communication link can be a wireless link. Examples of the wireless link can include a cellular communication link, a near field communication link, Wi-Fi, Bluetooth, a 60 GHz wireless link, or any wireless communication link for communication. Through the communication link, the communication interface 165 can transmit the following data to the console 110: the data indicates the determined position and / or orientation of the HWD 150, the determined user gaze direction, and / or hand tracking measurement results. In addition, through the communication link, the communication interface 165 can receive instructions from the console 110 or image data corresponding to the image to be rendered, as well as additional data associated with the image.

[0036] In some embodiments, the image renderer 170 includes an electronic component, or a combination of electronic components and software components, that generates one or more images for display (e.g., based on changes in the field of view in the artificial reality space). In some embodiments, the image renderer 170 is implemented as a processor (or a graphical processing unit (GPU)) that executes multiple instructions to perform the various functions described herein. The image renderer 170 may receive image data describing an artificial reality image to be rendered, and additional data associated with the image, through the communication interface 165, and may render the image through the electronic display 175. In some embodiments, the image data from the console 110 may be encoded, and the image renderer 170 may decode the image data to render the image. In some embodiments, the image renderer 170 receives object information and depth information in the additional data from the console 110, the object information indicating a virtual object in the artificial reality space, and the depth information indicating the depth of the virtual object (or the distance from the HWD 150). In one aspect, the image renderer 170 may perform shading, reprojection, and / or blending based on the artificial reality image from the console 110, object information, depth information, and / or updated sensor measurements from the sensor 155 to update the artificial reality image to correspond to the updated position and / or orientation of the HWD 150. Assuming that the user turns his head after the initial sensor measurements, the image renderer 170 may generate a small portion (e.g., 10%) of the image corresponding to the updated field of view within the artificial reality based on the updated sensor measurements, and append the portion to the image in the image data from the console 110 by reprojection, rather than reconstructing the entire image in response to the updated sensor measurements. The image renderer 170 may perform shading and / or blending on the additional edges. Thus, the image renderer 170 may generate an image of the artificial reality without reconstructing the image of the artificial reality based on the updated sensor measurements. In some embodiments, the image renderer 170 receives hand model data indicating the shape, position, and orientation of the hand model corresponding to the user's hand, and overlays the hand model on the artificial reality image. Such a hand model can be presented as visual feedback to allow the user to provide various interactions within the artificial reality.

[0037] In some embodiments, the electronic display 175 is an electronic component that displays an image. The electronic display 175 may be, for example, a liquid crystal display or an organic light emitting diode display. The electronic display 175 may be a transparent display that allows the user to see through. In some embodiments, when the HWD 150 is worn by the user, the electronic display 175 is located near the user's eyes (e.g., less than 3 inches). In one aspect, the electronic display 175 emits light or projects light toward the user's eyes according to the image generated by the image renderer 170.

[0038] In some embodiments, lens 180 is a mechanical component that changes the light received from electronic display 175. Lens 180 can magnify the light from electronic display 175 and correct the optical errors associated with the light. Lens 180 can be a Fresnel lens, a convex lens, a concave lens, a filter, or any suitable optical component that changes the light from electronic display 175. Although electronic display 175 is very close to both eyes, through lens 180, light from electronic display 175 can still reach the pupil, so that the user can see the image displayed by electronic display 175.

[0039] In some embodiments, compensator 185 includes electronic components, or a combination of electronic components and software components, that perform compensation to compensate for any distortion or aberration. In one aspect, lens 180 introduces optical aberrations (e.g., chromatic aberration), pincushion distortion, barrel distortion, etc. Compensator 185 can determine compensation (e.g., pre-distortion) to be applied to the rendered image from image renderer 170 to compensate for the distortion caused by lens 180, and apply the determined compensation to the image from image renderer 170. Compensator 185 can provide the pre-distorted image to electronic display 175.

[0040] In some embodiments, the console 110 is an electronic component, or a combination of electronic components and software components, that provides content to be rendered to the HWD 150. In one aspect, the console 110 includes a communication interface 115 and a content provider 130. These components can operate together to determine the field of view of the artificial reality corresponding to the position of the HWD 150 and the gaze direction of the user of the HWD 150 (e.g., the user's field of view (FOV)), and can generate image data indicating the artificial reality image corresponding to the determined field of view. In addition, these components can operate together to generate additional data associated with the image. The additional data can be information associated with presenting or rendering artificial reality in addition to the artificial reality image. Examples of additional data include hand model data, mapping information (or simultaneous localization and mapping (SLAM) data) for converting the position and orientation of the HWD 150 in the physical space to the virtual space, motion vector information, depth information, edge information, object information, etc. The console 110 can provide image data and additional data to the HWD 150 for presenting artificial reality. In other embodiments, the console 110 includes Figure 1 More, fewer, or more of the components shown may be included. Figure 1 In some embodiments, console 110 is integrated as part of HWD 150 .

[0041] In some embodiments, the communication interface 115 is an electronic component, or a combination of an electronic component and a software component, that communicates with the HWD 150. The communication interface 115 may be a counterpart of the communication interface 165, which communicates with the communication interface 115 of the console 110 via a communication link (e.g., a wireless link). Through the communication link, the communication interface 115 may receive data from the HWD 150 indicating the determined position and / or orientation of the HWD 150, the determined user gaze direction, and hand tracking measurements. In addition, through the communication link, the communication interface 115 may transmit image data describing the image to be rendered and additional data associated with the artificial reality image to the HWD 150.

[0042] The content provider 130 is a component that generates content to be rendered according to the position and / or orientation of the HWD 150. In some embodiments, the content provider 130 can generate content to be rendered in combination with the gaze direction of the user of the HWD 150 and the user interaction based on the hand tracking measurement results in the artificial reality. In one aspect, the content provider 130 determines the field of view of the artificial reality according to the position and / or orientation of the HWD 150. For example, the content provider 130 maps the position of the HWD 150 in the physical space to the position in the artificial reality space, and determines the field of view of the artificial reality space from the position in the mapped artificial reality space along the direction corresponding to the mapped orientation. The content provider 130 can generate image data of an image describing the field of view of the determined artificial reality space, and transmit the image data to the HWD 150 through the communication interface 115. The content provider 130 can also generate a hand model corresponding to the hand of the user of the HWD 150 according to the hand tracking measurement results, and generate hand model data indicating the shape, position and orientation of the hand model in the artificial reality space. In some embodiments, the content provider 130 may generate additional data associated with the image (the additional data includes motion vector information, depth information, edge information, object information, hand model data, etc.), and transmit the additional data together with the image data to the HWD 150 through the communication interface 115. The content provider 130 may encode the image data describing the image, and may transmit the encoded data to the HWD 150. In some embodiments, the content provider 130 generates image data periodically (e.g., every 11 ms) and provides the image data to the HWD 150.

[0043] Figure 2 1 is a schematic diagram of HWD 150 according to an example embodiment. In some embodiments, HMD 150 includes front rigid body 205 and strap 210. Front rigid body 205 includes electronic display 175 ( Figure 2 Not shown), lens 180 ( Figure 2 ), sensor 155, eye trackers 160A, 160B, communication interface 165, and image renderer 170. Figure 2 In the illustrated embodiment, the communication interface 165, image renderer 170, and sensor 155 are located within the front rigid body 205 and may not be visible to the user. In other embodiments, the HWD 150 may have a Figure 2 For example, the communication interface 165, the image renderer 170, the eye tracker 160A, 160B and / or the sensor 155 may be located in a different configuration than that shown. Figure 2 In some embodiments, HWD 150 may include multiple communication interfaces 165. Similarly, Figure 1 The console 110 may include multiple communication interfaces 115. As described in more detail below in Section B, one or more of the communication interfaces 115, 165 may be configured to selectively perform beamforming to optimize a communication channel between the console 110 and the HWD 150. Similarly, the console 110 and the HWD 150 may dynamically and intelligently switch between active and idle one or more communication interfaces 115, 165 to optimize a communication channel between the console 110 and the HWD 150.

[0044] B. Computing System

[0045] Various operations described herein may be implemented on a computer system. Figure 3 is a block diagram of a representative computing system 314 that may be used to implement the present disclosure. In some embodiments, Figure 1 The console 110, HWD 150, or both are implemented by a computing system 314. The computing system 314 can be implemented as, for example, a consumer device, such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, glasses, head mounted display), desktop computer, laptop computer; or the computing system can be implemented using a distributed computing device. The computing system 314 can be implemented to provide a VR experience, an AR experience, or an MR experience. In some embodiments, the computing system 314 may include conventional computer components, such as a processor 316, a storage device 318, a network interface 320, a user input device 322, and a user output device 324.

[0046] The network interface 320 can provide a connection to a wide area network (e.g., the Internet) to which a wide area network (WAN) interface of the remote server system is also connected. The network interface 320 can include a wired interface (e.g., Ethernet) and / or a wireless interface that implements various radio frequency (RF) data communication standards, such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, 5G, 60GHz, Long Term Evolution (LTE), etc.).

[0047] User input device 322 may include any device (or devices) via which a user may provide signals to computing system 314; computing system 314 may interpret these signals as indicating a particular user request or information. User input device 322 may include any or all of the following: a keyboard, a touch pad, a touch screen, a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a microphone, a sensor (e.g., a motion sensor, an eye tracking sensor, etc.), and the like.

[0048] User output device 324 may include any device through which computing system 314 can provide information to a user. For example, user output device 324 may include a display for displaying images generated by computing system 314 or delivered to computing system 314. The display may combine various image generation technologies with supporting electronic devices (e.g., digital-to-analog converters or analog-to-digital converters, signal processors, etc.), such as liquid crystal displays (LCDs), light-emitting diodes (LEDs) including organic light-emitting diodes (OLEDs), projection systems, cathode ray tubes (CRTs), etc. Devices that act as both input devices and output devices, such as touch screens, may be used. Output device 324 may be provided in addition to or in place of a display. Examples include indicator lights, speakers, tactile "display" devices, printers, etc.

[0049] Some embodiments include electronic components storing computer program instructions in computer-readable storage media (e.g., non-transient computer-readable media), such as microprocessors, storage devices, and memory. Many of the multiple features described in this specification may be implemented as a process: the process is specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processors, these program instructions cause the processor to perform the various operations indicated in these program instructions. Examples of program instructions or computer codes include machine codes and files, which are generated, for example, by a compiler, and the files include high-level codes executed by a computer, electronic components, or microprocessors using an interpreter. Through suitable programming, processor 316 can provide various functions for computing system 314, including any of the following: functions described herein as being executed by a server or client, or other functions associated with a message management service.

[0050] It will be understood that computing system 314 is illustrative and various variations and modifications may be made. The computer system used in conjunction with the present disclosure may have other capabilities not explicitly described herein. In addition, although computing system 314 is described with reference to a specific block, it should be understood that these blocks are defined for the convenience of description and are not intended to imply a specific physical arrangement of each component part. For example, different blocks may be located in the same facility, in the same server rack, or on the same motherboard. In addition, these blocks do not have to correspond to physically different components. Each block may be configured to perform a variety of operations (e.g., by programming a processor or providing a suitable control circuit system), and each block may or may not be reconfigurable depending on the manner in which the initial configuration is obtained. Each embodiment of the present disclosure may be implemented in various devices, including electronic devices implemented using any combination of circuits and software.

[0051] C. Systems and methods for streaming-based object recognition

[0052] Reference now Figure 4 , depicted is a network diagram 400 showing a network path between a device 402 and an identification server 404 according to an example implementation of the present disclosure. Some devices (e.g., device 402) may operate in an augmented reality (AR) space / virtual reality (VR) space. Such devices 402 may include mobile devices / smartphones, AR glasses, VR head-mounted viewers (headsets), etc. In some aspects, device 402 may be similar to or include the following components: These components are similar to those described in the above reference Figures 1 to 3 The head wearable display 150 / computing device 110, computing system 314, etc. are described.

[0053] In some embodiments, device 402 may be configured to provide AR / VR functionality to a user. Some AR / VR devices 402 may be configured to provide visual search functionality. The visual search functionality may involve device 402 capturing images (or videos) of the environment of device 402 and detecting various objects in the environment. Device 402 may analyze an object (e.g., to extract, detect, or otherwise identify one or more features of the object). Device 402 may include a transceiver 403 configured to send data to (e.g., wirelessly) and / or receive data from one or more remote sources (e.g., server 404). In some aspects, transceiver 403 may be similar to the above referenced device 402. Figure 1 and Figure 2 The wireless interfaces 115, 165 described above and / or Figure 3The network interface 320 described above. The device 402 may be configured to transmit the object coordinates and one or more features (e.g., as a feature vector or some other formatted / structured data identifying the one or more features) to the recognition server 404 (e.g., via the transceiver 403). The recognition server 404 may analyze the object coordinates and one or more features corresponding to the object, and may determine a recognition result, which the recognition server 404 transmits back to the device 402 for rendering at the device 402 (e.g., to the wearer / user of the device 402). Thus, the device 402 may receive the recognition result from the recognition server 404 (e.g., via the transceiver 403). For example, the recognition result may include an overlay provided near or relative to the object (e.g., above the object / adjacent to the object / near the object, etc.) (e.g., within the viewport of the display of the device 402). The recognition result may include information related to the object (e.g., the name of the object, the price of the object, the location of the object, where to buy the object, etc.), and potential actions performed in relation to the object (e.g., making a call related to the object, ordering the object, etc.), etc.

[0054] In various embodiments, particularly where device 402 operates in or through a cellular (e.g., 5G) space / network, device 402 may transmit, send, or otherwise provide information (e.g., object coordinates and one or more features) to one or more cellular towers / radio heads (RRHs) / base stations 406 (e.g., next generation nodes gNBs for 5G) (e.g., via transceiver 403). Base station 406 may forward, send, or otherwise provide information to one or more digital units or distributed units (DUs) 408(1) to 408(N) (which may be located at a corresponding base station 410 similar to base station 406). The digital unit 408 may transmit, send, forward, or otherwise provide information to a user plane function (UPF) 412, which in turn may provide the information to the identification server 404 (e.g., via a mobile edge computing (MEC) device 414 or unit, or via some other Internet-based channel or communication link). The identification server 404 may analyze the information, may determine / detect / generate / identify identification results, and may return these identification results for receipt by the device 402 (via the transceiver 403) (e.g., via the same network path used to send the information to the identification server 404 or via a different network path).

[0055] In order to facilitate the recognition server 404 to determine the recognition result, the device 402 can process one or more images of various objects in various aspects to identify various features that can be used to classify multiple objects in the image into multiple categories. Such categories can include, for example, the type of object, land and water, plane and mountain range, building and tree, etc. In some embodiments, the device 402 can generate a feature vector, which includes a series of features / feature sets / feature arrays / a group of features generated for a group of categories. Compared with the image itself, the data packet including the feature vector can have a smaller payload. However, as the number of categories considered by the device 402 when generating the feature vector increases, the payload increases accordingly.

[0056] Due to the limited processing power of device 402 (or a corresponding tethered device), it may be challenging for device 402 to detect a large number of objects simultaneously, or even a limited number of objects if a user moves device 402 rapidly (e.g., thereby rapidly changing the viewport of device 402). Furthermore, due to the limited processing power and limited storage of device 402, it may be challenging to scale the number of categories that device 402 considers when determining features of an object. As a result, a wearer / user of the device may be unable to locate a target object (e.g., an object about which the user desires to obtain information) in the viewport.

[0057] refer to Figure 5 , depicted is a graph showing the remaining computing resources of a device 402 relative to the number of objects in the environment of the device 402 according to an example embodiment of the present disclosure. Figure 5 As shown, as the number of objects in the environment increases (e.g., depending on the functions being performed locally on device 402 or by a tethered device), the remaining computing resources decrease accordingly. However, depending on the functions being performed locally on device 402, the remaining computing resources may decrease at different rates. Figure 5 , for device 402 to extract or otherwise determine features associated with an object, as the number of objects in the environment increases, the remaining computing resources decrease at an inversely proportional rate (e.g., a relatively linear decrease). This is because, as the number of objects increases, device 402 will determine the features of many objects, and thus use / spend / consume more and more computing resources to determine these features. On the other hand, for device 402 to encode a video feed, the remaining computing resources of device 402 will be reduced to a stable state (e.g., regardless of the number of objects in the environment). In other words, for device 402 to encode a video feed, device 402 may spend / use / consume computing resources at the front end, and regardless of the number of objects in the environment of device 402, the computing resources will tend to stabilize.

[0058] Reference now Figure 6 , depicted is a block diagram of a system 600 for streaming-based object recognition according to an example embodiment of the present disclosure. System 600 may include one or more devices (e.g., device 402) and one or more recognition servers (e.g., recognition server 404). As described above, device 402 may be an AR / VR device, including a mobile device, a smartphone, AR glasses, a VR head-mounted viewer, etc.

[0059] Device 402 may include an image collector 602 and a video collector 604. Image collector 602 may be or include any device, component, element, and / or hardware designed or configured to collect / acquire images of the environment (e.g., external environment) of device 402. Image collector 602 may include, for example, one or more cameras and hardware communicatively coupled to the one or more cameras, which is configured to control or otherwise operate the one or more cameras to collect multiple images of the environment. Similarly, video collector 604 may be or include any device, component, element, and / or hardware designed or configured to collect / acquire videos of the environment (e.g., external environment) of device 402. Video collector 604 may include, for example, one or more cameras and hardware communicatively coupled to the one or more cameras, which is configured to control or otherwise operate the one or more cameras to collect video data (e.g., live video feed) of the environment. In some embodiments, image collector 602 and video collector 604 may use or include the same one or more cameras. In some embodiments, the image collector 602 may be configured to collect images from video data collected by or via the video collector 604. In some embodiments, the image collector 602 may be configured to collect images separately from video data collected by or via the video collector 604.

[0060] The device 402 may include a feature extractor 606. The feature extractor 606 may be or include any device, component, element, and / or hardware that is designed or configured to detect, sense, or otherwise identify one or more objects in one or more images acquired via the image collector 602 and / or one or more objects in a video acquired via the video collector 604. In the event that the feature extractor 606 detects an object for the first time, the feature extractor 606 may be configured to detect, determine, or otherwise identify the coordinates of the object within the image where the object was first detected (or a timestamp associated with a segment of video data). The coordinates may include, for example, the coordinates of a bounding box assigned to the object (e.g., the bounding box defines or surrounds the object), the center of the bounding box, etc. When the object is detected in subsequent images / video data, or when the device 402 otherwise moves relative to the object, the feature extractor 606 may be configured to track changes in the pose / position / orientation of the device 402 relative to the coordinates of the object (e.g., using the sensor 155 of the device 402). In this regard, feature extractor 606 may be configured to determine / monitor / track the position of the object as device 402 moves around in the environment (eg, such that the position of the object relative to device 402 changes).

[0061] Feature extractor 606 may be configured to extract, determine, distinguish or otherwise identify one or more features associated with one or more objects in the environment. In some embodiments, feature extractor 606 may be configured to maintain, include or otherwise access various categories of different objects. These categories may correspond to different types of objects that may be located in a given environment. Feature extractor 606 may be configured to evaluate each object in each object detected against various categories to identify features corresponding to these objects. In some embodiments, feature extractor 606 may be configured to identify the features of an object as a binary value (e.g., presence or absence) of a corresponding category, a probability (e.g., a percentage or other digital representation of the possibility of satisfying a particular category), etc.

[0062] Device 402 may be configured to send, transmit, transfer, or otherwise provide object coordinates and one or more features corresponding to one or more objects to recognition server 404 (e.g., via transceiver 403). Device 402 may be configured to provide object coordinates and one or more features to recognition server 404, as described above with reference to Figure 4As described. The recognition server 404 may be configured to receive object coordinates and one or more features from the device 402. The recognition server 404 may include an image / video matching engine 610 (commonly referred to as "matching engine 610"). The matching engine 610 may be or include any of the following devices, components, elements, and / or hardware: the device, component, element, and / or hardware is designed or configured to determine that one or more features of an object (e.g., determined by the device 402) correspond to, are related to, or otherwise match features associated with one or more known objects. For example, the matching engine 610 may be configured to use the one or more features to perform a search in one or more databases 612, or otherwise cross-reference data from one or more databases 612. The one or more databases 612 may be configured to store, maintain, contain, or otherwise include data associated with various objects and features corresponding to these objects. In some embodiments, the matching engine 610 may be configured to identify one or more candidate objects from the one or more databases 612 based on the features of the object received from the device 402 meeting the matching criteria of one or more features of the candidate objects in the database 612. Matching engine 610 may be configured to identify, select, detect, or otherwise determine a matching object from the one or more candidate objects (eg, based on the most or greatest number of matching features, a calculated matching score or probability, etc.).

[0063] The matching engine 610 can be configured to determine (e.g., generate / output) a recognition result related to the object (e.g., using the matching object as a reference). For example, the matching engine 610 can be configured to determine the recognition result by retrieving the recognition result from a database 612 associated with the matching object, by performing a query on another database or data repository to obtain information related to the matching object, etc. The recognition result can include data corresponding to the object detected by the device 402. The recognition server 404 can be configured to communicate, transmit, send, or otherwise provide the recognition result to the device 402 for rendering to a user / wearer / viewer of the device 402 (e.g., by, using, via, a rendering engine 618 of the device 402).

[0064] In some embodiments, device 402 may be configured to determine whether the remaining computing resources of device 402 meet one or more threshold criteria (e.g., equal to and / or below a threshold, such as Figure 5). Device 402 may be configured to determine whether the remaining computing resources of device 402 meet the threshold criteria when device 402 extracts features of objects in the environment (e.g., simultaneously, synchronously, in parallel). As described in more detail below, in the case where device 402 determines that the remaining computing resources meet the threshold criteria, device 402 may be configured to transmit, transfer, send or otherwise provide viewport coordinates and video data (e.g., a video feed of the environment (e.g., a live video feed), encoded video data, etc.). Device 402 may be configured to transmit, send or otherwise provide encoded video data and viewport coordinates to recognition server 404. In this regard, in the case where the remaining computing resources meet the threshold criteria, device 402 may be configured to transfer feature extraction to recognition server 402 by providing encoded video data to recognition server 404. Such implementations and embodiments may provide better resource allocation of device 402 and an improved user experience, as well as better power savings on device 402.

[0065] Reference now Figure 6 and Figure 7 , the device 400 may be configured to perform streaming-based object recognition. Specifically, Figure 7 A flowchart showing a method 700 for object recognition based on streaming according to an example embodiment of the present disclosure is depicted. The method 700 may be as described above with reference to Figure 4 and Figure 6 The device 402 is described to perform.

[0066] Method 700 may begin at step 702, where device 402 may initiate a video search. Video search may be or include an application, feature, or function provided by device 402 to a user / wearer that identifies an object in the environment of device 402 and renders, displays, or otherwise provides information related to the identified object (e.g., via display 175 of device 402). Device 402 may initiate a video search in response to a user / wearer of device 402 initiating a video search application or resource on device 402 (or a tethered device / paired device / communicatively coupled device). Device 402 may be configured to initiate a video search by acquiring video data or image data of the environment of device 402. Device 402 may be configured to acquire video data or image data using image collector 602 and / or video collector 604 of device 402 and one or more sensors 155 (e.g., one or more cameras of device 402). In some embodiments, device 402 may be configured to capture images (eg, via image collector 602 ) from a video feed or video stream generated via video collector 604 .

[0067] In step 704, device 402 may be configured to identify multiple objects included in one or more images. These objects may be located in the environment of device 402. When device 402 moves around in the environment, device 402 may be configured to collect, generate or otherwise obtain images of multiple objects located in the environment. Device 402 may be configured to obtain images of objects located in the viewport of device 402. As described herein, the viewport may generally include a portion or region of the environment, currently viewed or displayed on device 402 (e.g., at its display 175). When device 402 moves around in the environment (e.g., and camera sensor 155 obtains the image of the environment), the viewport may change accordingly. When the viewport changes over time, different objects located in the environment may move in or out of the viewport over time.

[0068] Device 402 may be configured to detect or identify an object located within a viewport. In some embodiments, feature extractor 606 may be configured to detect an object located in a viewport. Feature extractor 606 may be configured to use, utilize, or otherwise apply various neural network-based methods or non-neural network-based methods to detect or identify objects within an environment. For example, neural network-based methods may include, for example, region proposals (e.g., regional convolutional neural networks (R-CNN), fast R-CNN, faster R-CNN, cascaded R-CNN), single-shot multi-frame detectors, you only look once, single-shot refinement neural networks for object detection (including RefineDet), retina networks (retina-net), deformable convolutional networks, and / or any other neural network-based object detection systems). Examples of non-neural network methods may include, for example, a Viola-Jones object detection framework based on Haar features, scale-invariant feature transform (SIFT), histogram of oriented gradient (HOG) features, or any other non-neural network based object detection system. Since these services are typically time-limited, one or a combination of these methods may be used to return results within the time limit, which may also be performed by considering the transmission capacity of the network connecting the device to the server. The feature extractor 606 may be configured to apply one or more of the aforementioned object detection methods to the image to identify or detect an object located in the viewport. In the case where the feature extractor 606 identifies / detects an object in the viewport (or otherwise identifies / detects an object located in one or more images obtained by the device 402), the feature extractor 606 may be configured to identify coordinates associated with the object. These coordinates may be or include relative coordinates of the object (e.g., relative to the local coordinate system of the device 402).

[0069] In step 706, device 402 may be configured to determine, detect, identify or otherwise extract one or more features of an object detected in the environment of device 402. In some embodiments, feature extractor 606 may be configured to maintain various object classification models or classifiers associated with different object features. Feature extractor 606 may be configured to apply object classification models or classifiers to one or more images of a given object to extract features corresponding to the object. Feature extractor 606 may be configured to generate, detect, determine or otherwise identify a feature set corresponding to an object. In some embodiments, a feature set may include multiple features that can be used to identify an object. A feature set may be, for example, a feature vector. Feature extractor 606 may be configured to generate a feature set for the object based on the output / results from one or more classification models / classifiers applied to an image of a given object.

[0070] In some embodiments, device 402 may be configured to extract features of each of the objects identified in step 704. Device 402 may be configured to extract features when a new object is detected or identified in the environment. Thus, in some embodiments, device 402 may be configured to loop between steps 704 and 706.

[0071] Device 402 may be configured to determine, detect, or otherwise identify a computing resource status of device 402. Device 402 may be configured to identify a computing resource status of device 402 when device 402 identifies objects in step 704 and extracts features of these objects in step 706. Device 402 may be configured to determine the computing resource status as remaining (available) computing resources. For example, remaining computing resources may be or include a utilization percentage of a central processing unit (CPU) of device 402, remaining memory, the number of processes that device 402 can execute, etc. Device 402 may be configured to compare the computing resource status with a threshold. The threshold may be, for example, a threshold remaining computing resource (e.g., a threshold utilization percentage, a threshold remaining memory, a threshold number of processes that device 402 can execute, etc.).

[0072] In step 708, the device 402 may be configured to determine whether the computing resource status meets the threshold criteria based on the comparison. For example, where the computing resource status is a utilization percentage, the computing resource status may meet the threshold criteria in response to the utilization percentage exceeding (or meeting) the threshold. As another example, where the computing resource status is the remaining memory, the computing resource status may meet the threshold criteria in response to the remaining memory reaching or falling below the threshold. As yet another example, where the computing resource status is the number of processes that can be executed by the device 402, the computing resource status may meet the threshold criteria in response to the number of executable processes reaching or falling below the threshold.

[0073] At step 708, if / when the device 402 determines that the computing resource criteria are not met (e.g., the device has sufficient computing resources to perform object detection and feature extraction locally), the device 402 may be configured (e.g., at step 716) to transmit, send, communicate, or otherwise provide (e.g., via the transceiver 403) the object coordinates of the object and the identified features to the recognition server 604. The device 402 may be configured to transmit, send, communicate, or otherwise provide (e.g., via the transceiver 403) the object coordinates of the object and the identified features to the recognition server 604. Figure 4 The network path shown in FIG. 7 provides the object coordinates and the identified features (e.g., as a feature set, feature vector, etc.) to the recognition server 604. As described in more detail below with respect to step 718, the recognition server 604 can be configured to recognize the object (e.g., based on the features matching or substantially matching a candidate object from the database 612), and can transmit a recognition result associated with the matching object to the device 402.

[0074] At step 708, if / when the device 402 determines that the computing resource criteria are met, the method 700 may proceed to step 710. At step 10, the device 402 may be configured to determine, negotiate, set, or otherwise establish one or more parameters for transmitting video data to the recognition server 404. Parameters that may be selected, set, established, or otherwise determined for a session (e.g., by the device 402 and / or the recognition server 404) may include, for example, bit rate, resolution, encoding scheme, and compression. In some embodiments, the device 402 and the recognition server 404 may be configured to establish one or more parameters for a session between the device 402 and the recognition server 404. The device 402 and the recognition server 404 may be configured to negotiate the one or more parameters prior to establishing the session between the device 402 and the server 404. For example, in some embodiments, the device 402 may be configured to transmit a session description protocol (SDP) packet or proposal that includes one or more parameters selected by the device 402 for the session. The recognition server 404 may be configured to receive the SDP packet, identify one or more parameters selected by the device 402, and generate an SDP response. The SDP response may include one or more parameters selected by the recognition server 404 based on one or more parameters from the SDP packet. For example, where the recognition server 404 agrees, agrees, or otherwise accepts one or more parameters selected by the device 402 from the SDP packet, the recognition server 402 may generate an SDP response including these same one or more parameters. On the other hand, where the recognition server 404 identifies one or more parameters that are considered unacceptable, the recognition server 404 may select different parameters. The device 402 and the recognition server 404 may be configured to determine one or more parameters of the video session based on communications / packets / messages exchanged between the device 402 and the recognition server 404.

[0075] In step 712, device 402 may be configured to generate video data. Video data may be or include a video stream, video feed, etc. of the environment of device 402. Video data may be or include live video data. Device 402 may be configured to generate video data from, based on, or using the following data: the data is collected by video collector 604 using one or more camera sensors 155 of device 402. In some embodiments, device 402 may be configured to generate video data based on or according to one or more parameters set / established / determined for a session. For example, in the case where each parameter includes a compression or encoding scheme, device 402 may be configured to compress and / or (e.g., using video encoder 608) encode video data according to the negotiated / determined / set / established parameters. Similarly, in the case where the parameter includes resolution, video encoder 608 may be configured to generate video data with the established resolution parameters (e.g., by enlarging or reducing the resolution according to the established resolution parameters). Additionally, where the parameters include a bit rate, the video encoder 608 may be configured to generate video data for transmission, compression, encoding, etc. at the established bit rate.

[0076] Device 402 may be configured to determine, detect or otherwise identify the viewport of device 402. In some embodiments, video collector 604 may be configured to determine the viewport (e.g., the border, range and / or coordinates of the viewport). Device 402 may be configured to determine the viewport based on the current location, posture, position or orientation of device 402 relative to the environment. In some embodiments, the viewport may be or include a portion or region of the environment that is currently being displayed to the user (e.g., on display 175). Device 402 may be configured to determine coordinates associated with or otherwise corresponding to the viewport. Device 402 may be configured to determine the position, location, posture or orientation of device 402 (e.g., camera sensor 155) relative to a coordinate system. Device 402 may be configured to use camera sensor 155 and / or various other sensors of device 402 (e.g., inertial measurement unit (IMU), one or more accelerometers, etc.) to determine the orientation of device 402.

[0077] In some embodiments, device 402 may be configured to determine coordinates associated with a local coordinate system of device 402. For example, device 402 may be configured to define a local coordinate system of device 402, wherein device 402 is located at the center of the local coordinate system (e.g., [0, 0, 0]). A viewport may be defined as an area corresponding to the direction in which the camera sensor 155 of device 402 is facing (e.g., collecting images / videos). For example, the direction of the camera sensor 155 (e.g., the outward direction) may be defined as a vector extending through the center of the viewport. Device 402 may be configured to determine viewport coordinates based on the orientation of device 402 and a fixed frame corresponding to the size of the viewport. For example, in the case where device 402 determines the center of the viewport (e.g., based on the direction of the camera sensor 155) and a fixed frame corresponding to the size of the viewport, device 402 may be configured to use the coordinates of the vector defined for camera sensor 155 and the fixed frame size to determine the coordinates of the viewport. As device 402 moves around in an environment, device 402 may be configured to update, revise, modify, or otherwise determine new coordinates of the viewport.

[0078] At step 714, device 402 may be configured to transmit, send, communicate, or otherwise provide the viewport coordinates and the video data (e.g., generated at step 712) to recognition server 404 (e.g., via transceiver 403). In some embodiments, device 402 may be configured to transmit, send, communicate, or otherwise provide the viewport coordinates and the video data (e.g., generated at step 712) to recognition server 404 (e.g., via transceiver 403). Figure 4 The network path described above is used to provide video data and viewport coordinates. In some embodiments, device 402 can be configured to stream the video data generated in step 712 to recognition server 404 (e.g., via transceiver 403). Device 402 can be configured to send viewport coordinates at various time intervals / time instances when device 402 (e.g., based on motion data of device 402) determines that the viewport has changed. In some embodiments, device 402 can be configured to transmit viewport coordinates at a first time instance and transmit any determined updates to the viewport at a different time instance (e.g., as an increment or change relative to the coordinates sent at the first time instance).

[0079] At step 718, device 402 may be configured to receive the recognition result from recognition server 404 (eg, via transceiver 403). Fig. 9Discussed in more detail determining and sending recognition results.In step 720, device 402 can be configured to determine whether there are any other objects to be identified in the environment.In the case where there are no other objects to be identified, device 402 can end video search (step 722).On the other hand, in the case where there are other objects that are not identified in the environment, method 700 can (for example, when the computing resource standard was not previously met) loop back to step 706, or (for example, when the computing resource standard is met) loop back to step 712.

[0080] Reference now FIG. 8A to FIG. 8D , depicted is a series of views of an environment 800 of a device 402 according to an example implementation of the present disclosure. Fig. 8A As shown, the environment 800 may include various objects (e.g., object (1), object (2), object (3), etc.). The device 402 may be configured to capture various representations (e.g., video feeds, images, etc.) of the environment 800 including multiple objects (e.g., using one or more camera sensors 155 via the image collector 602 and / or the video collector 604). The device 402 may be configured to detect or identify each object located in the environment 800, and may identify coordinates associated with each object. For example, the device 402 may be configured to identify coordinates associated with a bounding box of a corresponding object. In some instances (as described above with reference to steps 706 and 716), the device 402 may identify features of the object, and may transmit the object coordinates and one or more features to the recognition server 404. On the other hand, as described above with reference to steps 710 to 714, in some instances, the device 402 may establish video session parameters, encode video data, and transmit the coordinates of the viewport to the recognition server 404.

[0081] like FIG. 8A to FIG. 8D As shown, the viewport may contain, enclose, or include a portion or region of environment 800 that is currently being displayed to a user by device 402 (e.g., on display 175). Device 402 may determine the viewport coordinates based on a fixed frame size of the viewport and an orientation of device 402 (e.g., using data from one or more of the device's various sensors 155). FIG. 8A to FIG. 8D As shown in the process above, when the orientation of device 402 changes over time, the viewport may change accordingly over time.

[0082] exist Fig. 8AAt the first time instance (t1) shown, the device 402 can detect or identify an object located in the environment 800. The device 402 can be configured to establish, detect, identify, or otherwise determine the coordinates of each of the respective objects. The device 402 can be configured to establish, for each object, a bounding box that surrounds, contains, or otherwise defines the object. The device 402 can be configured to transmit video data and viewport coordinates to the recognition server 404 to receive information about the object rendered to the user / wearer of the device 402 on the display 175. In some instances, the device 402 can transmit video data to the recognition server 404 in / on the session / as part of the session between the device 42 and the recognition server 402. The recognition server 402 can be configured to extract or otherwise identify features of objects located in the environment, and can provide information related to the object back to the device 402 on the session. As described below with reference to Fig. 9 As described in more detail, in some embodiments, the recognition server 402 can be configured to provide information / data related to the object in order based on the coordinates of the object relative to the viewport. As described in more detail below, such an implementation can provide a priority ordering of more relevant object-related information to be sent to the device 402 earlier.

[0083] Now refer to FIG. 8A to FIG. 8D For reference Fig. 9 , depicted is a flow chart showing a method 900 for transmitting recognition results according to an example embodiment of the present disclosure. Method 900 may be performed by a recognition server 404 using data (e.g., viewport coordinates, video data, etc.) received from a device 402. As described above, the recognition server 404 may be configured to receive various updates to the viewport coordinates from the device 402 at various intervals (e.g., at fixed intervals, in response to viewport changes, etc.). As a brief overview, method 900 may begin at step 902. At step 904, the recognition server 404 may update the position of the viewport. At step 906, the recognition server 404 may retrieve / generate recognition results. At step 908, the recognition server 404 may determine one or more distances from one or more objects to the center of the viewport. At step 910, the recognition server 404 may arrange the objects (or recognition results of the objects) in order based on the one or more distances. At step 912, the recognition server 904 transmits the recognition results.

[0084] Method 900 may begin at step 902. In some embodiments, method 900 may begin when recognition server 404 receives first video data and viewport coordinates from device 402. In some embodiments, device 402 may send a request for / in such a manner as Fig. 8A800. The device 402 may include first video data and viewport coordinates for a first time instance (t1) as shown. The video data may include representations of objects (1) to (3) located in the environment 800 of the device. In some embodiments, the device 402 may send the first video data, the viewport coordinates of the viewport at the first time instance, and the object coordinates of one or more objects detected by the device 402 at the first time instance. In some embodiments, the device 402 may send the first video data and the viewport coordinates at the first time instance, and the recognition server 404 may determine the object coordinates of the objects represented in the video data at the first time instance.

[0085] At step 904, the recognition server 404 may update the position (e.g., orientation, extent, boundaries) of the viewport. In some embodiments, the recognition server 404 may update the position of the viewport based on the new / updated / second viewport coordinates received from the client (e.g., at a second time instance (t2)). Figure 8B As shown, the viewport may change between a first time instance and a second time instance (e.g., based on a user / wearer of device 402 moving the device, or based on a scene / object change / movement in the viewport). Device 402 may transmit, send, convey, or otherwise provide updated viewport coordinates for the second time instance to recognition server 404. In some embodiments, the updated viewport coordinates may be an increment, shift, or change relative to the first viewport coordinates. Recognition server 404 may receive changes originating from the first viewport coordinates and shift / modify / change / update the viewport coordinates based on the changes received from device 402. In some embodiments, the updated viewport coordinates may include a new set of coordinates for the second time instance that is different from the first set of coordinates received from device 402. Recognition server 404 may update the viewport coordinates by replacing the first viewport coordinates with the second viewport coordinates.

[0086] At step 906, the recognition server 404 may retrieve (e.g., produce / provide / generate) recognition results. The recognition server 404 may retrieve recognition results for each of a plurality of objects included in the video data from the device 402. In some embodiments, the recognition server 404 may retrieve recognition results by detecting, determining, identifying, or otherwise extracting one or more features corresponding to a given object. In some embodiments, the matching engine 610 may parse / analyze / process image data or video data including an object to identify one or more features corresponding to the object. The matching engine 610 may be configured to perform the matching operation in a manner similar to that described above with reference to FIG. Figure 6The matching engine 610 may identify one or more features in a manner similar to the feature extractor 606 described above. The matching engine 610 may use the one or more features of the object and the one or more databases 612 to identify a matching object corresponding to the object in the image data / video data. For example, the matching engine 610 may identify one or more candidate objects from the one or more databases 612 that have one or more features that are included in the one or more databases 612 and match / substantially match / correspond to the one or more features of the object. The matching engine 610 may select or identify a matching object from the candidate objects based on a matching score or selection criteria for matching the object in the image data / video data with one or more objects from the one or more databases 612. For example, the matching score / selection criteria may include a weighted number of matching features, a threshold number of matching features, etc.

[0087] Once the recognition server 404 identifies the matching object from one or more databases 612, the matching engine 610 can determine, generate or otherwise identify the recognition result corresponding to the matching object. In some embodiments, one or more databases 612 may include the following recognition results: these recognition results are linked to the data entry corresponding to the matching object or are otherwise associated with the data entry. In some embodiments, the matching engine 610 can identify the recognition result in the following manner: use the identifier corresponding to the matching object or other data to perform a search, search or other query to retrieve or receive the recognition result. The recognition result may include, for example, information related to the object (e.g., object name, telephone number associated with the object, location or address for purchasing the object, etc.), actions that can be performed in relation to the object (e.g., calling or sending a message / email, ordering, etc.).

[0088] At step 908, the recognition server 404 may determine one or more distances from the one or more objects to the center of the viewport. In the event that multiple objects are located in the environment 800, the recognition server 404 may determine the distance of each of the multiple objects to the center of the viewport. In some embodiments, the action ranking engine 614 of the recognition server 404 may determine the one or more distances from the one or more objects to the center of the viewport. The action ranking engine 614 may be or include any device, component, element, or other hardware configured or designed to rank objects based on the actions of the wearer / user of the device 402. Fig. 8A and Figure 8BAs shown in the process between , the wearer / user of device 402 may have moved the device toward the third object. Action sequencing engine 614 can determine the distance of each of the objects in environment 600 (e.g., coordinates associated with or corresponding to each of the objects) to a reference point (e.g., a center or focal point) of the viewport. Action sequencing engine 614 can use the latest (e.g., most recent) viewport coordinates (e.g., coordinates used to update the viewport in step 904) to determine the distance of each of the objects to the reference point. FIG. 8A to FIG. 8D In the example shown, it is assumed that the recognition server 404 first receives Fig. 8A , and subsequently receives the video data of the first time instance (t1) shown in Figure 8B If the updated viewport coordinates at the second time instance (t2) are shown in , the recognition server 404 can use the viewport coordinates at the second time instance (t2) to determine the distance of the object from the viewport reference point.

[0089] In step 910, the recognition server 404 may sort the objects in order based on one or more distances. In some embodiments, the recognition server 404 may sort the recognition results of the objects in order based on the determined distances of the corresponding objects from the reference point of the current viewport. In some embodiments, the priority engine 616 of the recognition server 404 may sort the recognition results of the objects in order. The priority engine 616 may be or include any device, component, element, or other hardware that is configured or designed to generate, establish, produce, organize, or otherwise provide recognition results in order or sequence for transmission or delivery to the device 402. In some embodiments, the priority engine 616 may provide recognition results in order or sequence based on which objects are closest to the reference point of the viewport. For example, where the reference point is the center of the viewport, the priority engine 616 may provide recognition results in order or sequence based on which objects are closest to the center of the viewport. Figure 8B As shown, the second object may be closest to the center of the viewport, followed by the third object, where the first object is farthest from the center of the viewport. In this example, the priority engine 616 can provide recognition results in the order or sequence of object (2), object (3), and object (1).

[0090] In step 912, the recognition server 904 transmits the recognition results. In some embodiments, the recognition server 904 can transmit the recognition results to the device 402. The recognition server 904 can transmit the recognition results to the device in the order or sequence determined in step 910. In this way, the device 402 can receive the recognition results in an order or sequence based on the distance of the object to the center of the viewport (or other reference point). The rendering engine 618 of the device 402 can receive the recognition results from the recognition server 904 (e.g., in the order or sequence in which the recognition results were sorted in step 910). The rendering engine 618 can render or display the recognition results to the user or wearer of the device 402 (e.g., via the display 175). In some embodiments, the rendering engine 618 can render the recognition results for a subset of the objects based on one or more objects located within the viewport. For example, Figure 8C As shown, the rendering engine 618 can render the recognition results of the second object and the third object (e.g., the coffee machine and John Smith) based on the second object and the third object being located within the viewport at the third time instance (t3). On the other hand, since the first object (e.g., the laptop) is not located within the viewport, the rendering engine 618 can abandon / bypass / skip rendering the recognition results. However, as Fig.8D As shown, when the first object (eg, at a fourth time instance (t4)) moves into the viewport, the rendering engine may render the recognition result of the fourth object (eg, a laptop computer).

[0091] When new video data is received from device 402 and when the viewport of device 402 changes (e.g., based on the movement of device 402), method 900 can loop back to step 904. In this way, device 402 can stream video data and viewport coordinates to recognition server 404, and recognition server 404 can identify objects in the video data and transmit data corresponding to each identified object back to device 402 in an order, priority, or sequence based on the location of the object from the most recent (e.g., nearest) viewport reference point. Such an implementation can provide more relevant / important / relevant information received by the device for display earlier, which improves the overall user experience and reduces the perceived latency of visual search deployment on the device.

[0092] Reference now Fig.10 , depicted is a flowchart showing a method 1000 for object recognition based on streaming according to an exemplary embodiment of the present disclosure. The method 1000 may be referred to above. Figures 4 to 6 As a brief overview, at step 1002, the device transmits video data and a first coordinate. At step 1004, the device transmits a second coordinate. At step 1006, the device receives data corresponding to each object. At step 1008, the device renders the data.

[0093] In step 1002, the device transmits video data and a first coordinate. In some embodiments, the device transmits video data and a first coordinate to a server (e.g., identification server 402). The device may operate, use, or otherwise control a transceiver of the device to transmit video data and a first coordinate to the server. In some embodiments, the first coordinate may be the coordinate of a first viewport of the device at a first time instance. The video data may include various objects (e.g., objects represented in an image / video rendered in the first viewport) located in the first time instance. In some embodiments, the device may determine the first coordinate based on or according to sensor data of the device. The sensor data may be, for example, an inertial measurement unit (IMU) or an accelerometer of the device. The sensor data may also include camera data from one or more cameras of the device. For example, the device may determine the first coordinate using a fixed frame size corresponding to the viewport size and the direction facing the device 402. The device may use a camera (e.g., a camera's viewing angle), an IMU / accelerometer, etc. to determine the direction facing the device.

[0094] In some embodiments, the device can establish one or more parameters for processing video data or transmitting video data to a server. The device can establish / configure / set one or more parameters with the server. The one or more parameters may include, for example, the transmission, encoding, compression, etc. of video data. The device can negotiate with the server to establish one or more parameters. For example, the device can negotiate with the server by exchanging various session description protocol (SDP) packets or transmissions between the device and the server to set, determine or otherwise establish one or more parameters. The device can provide video data according to the one or more established parameters. For example, the device can encode and / or compress video data according to any encoding / compression parameter, the device can process (e.g., zoom in or out) video data according to any resolution parameter, and the device can transmit video data at a bit rate according to any bit rate parameter, etc.

[0095] At step 1004, the device transmits a second coordinate. In some embodiments, the device may transmit a second coordinate of a second viewport of the device at a second time instance. The device may use a transceiver of the device to transmit the second coordinate. In some embodiments, the device may determine that the second coordinate is transmitted in response to detecting movement or motion of the device (e.g., indicating a viewport change). In some embodiments, the device may determine that the second coordinate is transmitted in response to a fixed interval. In other words, the device may transmit the second coordinate periodically, or transmit the second coordinate on demand in response to detecting a viewport change. The device may determine the coordinates of the second viewport based on motion data from one or more various sensors of the device, based on data from one or more cameras of the device, and the like.

[0096] In step 1006, the device receives data corresponding to each object. In some embodiments, the device may receive data corresponding to one or more objects within the second viewport. The device may receive the data using or via a transceiver of the device (e.g., for transmitting video data and viewport coordinates). The device may receive data from a server (e.g., where the device sent viewport coordinates and video data). The data may be received in order based on the coordinates of the one or more objects relative to the second viewport. In some embodiments, and as described above with reference to Fig. 9 As described, the recognition server can sequentially establish or generate data corresponding to each object based on the position or coordinates of each object relative to the second viewport (e.g., relative to a reference point of the second viewport, such as the center or center point of the second viewport). The device can receive objects sequentially set or provided by the recognition server.

[0097] In some embodiments, at a second time instance, two or more objects may be located within the viewport. For example, a first object may be located at a first distance from a reference point, and a second object may be located at a second distance from the reference point (e.g., where the second distance is greater than the first distance). The recognition server may send data corresponding to the first object before sending data corresponding to the second object to the device. In some embodiments, the recognition server may send data corresponding to the first object before sending data corresponding to the second object by sending a first data packet of first data of the first object and a second (subsequent) data packet of second data of the second object. In some embodiments, the recognition server may send a single data packet or stream, but the first data may be represented in the data packet or stream before the second data. The device may receive data for the first object before receiving data for the second object from the server.

[0098] In step 1008, the device renders the data. In some embodiments, the device may render data related to one or more objects within a third viewport at a third time instance. The device may render data related to the one or more objects by annotating / overlaying the data on or near the corresponding one or more objects. The device may render data related to the one or more objects by annotating a bounding box that surrounds or surrounds the corresponding one or more objects. The device may render data related to the one or more objects by annotating a portion or region that is spatially close to the corresponding one or more objects. The device may render data related to the one or more objects on a display that displays a third viewport (including the corresponding one or more objects) at the third instance time.

[0099] In some embodiments, when the device moves within the environment, the object may move in and out of the viewport. For example, an object that exists or is located in the first viewport at a first time instance may not exist or be located in the second viewport at a second time instance. The device may receive data corresponding to the object from the server (e.g., after, after, or after receiving any data of the object located in the second viewport). The device may determine that the object is located in a third viewport (e.g., the device has moved so that the object is now located in the third viewport). The device may render data related to the object in the third viewport. Similarly, as another example, an object that exists or is located in the second (or even the first) viewport may not be located in the third viewport. The device may determine that the object is located or positioned outside the viewport. The device may abandon / bypass rendering data corresponding to the object in the third viewport. However, if at a fourth time instance, the device determines that the object moves from outside the third viewport to being located in the fourth viewport, the device may render data related to the object in the fourth viewport.

[0100] In some embodiments, the device may determine whether the available (e.g., remaining) computing resources of the device meet a threshold criterion. The device may determine whether the available computing resources of the device meet a threshold criterion while the device detects objects within the viewport at various time instances and extracts features associated with the objects. For example, and as described above with reference to Figure 7 As described, in the case where the device (e.g., at the fourth time instance) determines that the available computing resources of the device meet the threshold criteria (e.g., the available computing resources or the remaining computing resources are greater than or equal to the threshold), the device can detect the object from the video data, determine (or detect, identify, or otherwise extract) the features of the object, and transmit the coordinates and one or more features of the object to the server. However, in the case where the device determines that the available computing resources of the device do not meet the threshold criteria, the device can transmit the video data and the coordinates of the viewport to the server, as described above with reference to step 1002. At this point, the device can perform streaming-based object recognition in the following manner: in response to the available computing resources or the remaining computing resources not meeting the threshold criteria for feature extraction locally on the device (or through a tethered device / paired device), the video data and the viewport coordinates are transmitted to the server.

[0101] Now that some illustrative embodiments have been described, it is apparent that the foregoing, which has been presented by way of example, is illustrative and not restrictive. Specifically, although many of the examples presented herein involve specific combinations of method actions or system elements, these actions and these elements may be combined in other ways to achieve the same purpose. The actions, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from other embodiments or similar roles in the embodiments.

[0102] The hardware and multiple data processing components for implementing the various processes, operations, illustrative logic, logic blocks, modules and circuits described in conjunction with the multiple embodiments disclosed herein can be implemented or executed using the following components designed to perform the functions described herein: a general-purpose single-chip processor or a general-purpose multi-chip processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, or can be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented as a combination of multiple computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some embodiments, specific processes and methods can be performed by circuits for given functions. The memory (e.g., memory, storage unit, storage device, etc.) may include one or more devices (e.g., random access memory (RAM), read-only memory (ROM), flash memory, hard disk storage, etc.) for storing data and / or computer code, which is used to complete or facilitate the various processes, layers and modules described in the present disclosure. The memory may be or include volatile memory or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. According to an exemplary embodiment, the memory is communicatively connected to the processor via a processing circuit and includes computer code for executing (e.g., by a processing circuit and / or a processor) one or more processes described herein.

[0103] The present disclosure contemplates various methods, systems, and program products on any machine-readable medium for implementing various operations. The various embodiments of the present disclosure may be implemented using existing computer processors, or may be implemented by a dedicated computer processor for a suitable system combined for this purpose or other purposes, or may be implemented by a hard-wired system. The various embodiments within the scope of the present disclosure include multiple program products, which include machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media may be any available medium that can be accessed by a general-purpose computer or a special-purpose computer, or other machines with a processor. As an example, such machine-readable media may include RAM, ROM, erasable programmable read-only memory (EPROM), electronic erasable read-only memory (EEPROM), or other optical disk storage, disk storage, or other magnetic storage devices, or any other medium as follows: the other medium may be used to carry or store the desired program code in the form of machine-executable instructions or data structures, and may be accessed by a general-purpose computer or a special-purpose computer, or other machines with a processor. The above combinations are also included in the scope of machine-readable media. Machine-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

[0104] The words and terms used herein are for descriptive purposes and should not be considered limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized by," and variations thereof herein is meant to include the items listed thereafter, equivalents of those items, and additional items, as well as alternative embodiments consisting exclusively of the items listed thereafter. In one embodiment, the systems and methods described herein consist of one of the described elements, actions, or components, of various combinations of more than one of the described elements, actions, or components, or of all of the described elements, actions, or components.

[0105] Any reference to the embodiments or elements or actions of the systems and methods mentioned herein in the singular may also encompass embodiments including a plurality of these elements, and reference to any embodiments or elements or actions herein in the plural may also encompass embodiments including only a single element. References in the singular or plural are not intended to limit the systems or methods of the present disclosure, their parts, actions or elements to singular or plural configurations. References to any actions or elements based on any information, action or element may include embodiments based at least in part on any information, action or element.

[0106] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment or example. These terms as used herein do not necessarily all refer to the same embodiment. Any embodiment may be combined, inclusively or exclusively, with any other embodiment in any manner consistent with the various aspects and embodiments disclosed herein.

[0107] Where reference numerals are attached to technical features in the drawings, detailed description or any claims, these reference numerals have been included to increase the intelligibility of the drawings, detailed description and claims. Therefore, the presence or absence of these reference numerals will not have any limiting effect on the scope of any claim element.

[0108] Without departing from the characteristics of the systems and methods described herein, the systems and methods described herein may be implemented in other specific forms. Unless otherwise expressly noted, references to "approximately," "about," "substantially," or other degree terms include changes of plus or minus 10% (+ / -10%) from a given measurement, unit, or range. The coupled elements may be electrically coupled, mechanically coupled, or physically coupled to each other directly or through intermediate elements. Therefore, the scope of the systems and methods described herein is indicated by the appended claims rather than the preceding description, and changes within the meaning and scope of the equivalents of the claims are included therein.

[0109] The term "coupled" and variations thereof include two members being directly or indirectly connected to one another. Such connection may be static (e.g., permanent or fixed) or movable (e.g., removable or releasable). Such connection may be achieved by two members being directly coupled to one another or coupled to one another, wherein the two members are coupled to one another using a separate intermediate member, and coupled to one another through any additional intermediate member, or wherein the two members are coupled to one another using an intermediate member that is integrally formed as a single, unitary body with one of the two members. If "coupled" or variations thereof are modified by additional terms (e.g., directly coupled), the general definition of "coupled" provided above is modified by the plain language meaning of the additional terms (e.g., "directly coupled" means that the two members are connected without any separate intermediate member), resulting in a narrower definition than the general definition of "coupled" provided above. Such coupling may be mechanical, electrical, or fluidic.

[0110] References to "or" may be interpreted as inclusive, such that any term described using "or" may mean any of a single term, more than one term, and all of the terms described. References to "at least one of 'A' and 'B'" may include only 'A', only 'B', and both 'A' and 'B'. Such references used in conjunction with "including" or other public terms may include additional items.

[0111] Without departing substantially from the teachings and advantages of the subject matter disclosed herein, modifications may be made to the elements and actions described, such as changes in the size, dimensions, structure, shape and proportion of the various elements, parameter values, mounting arrangement structures, use of materials, colors, orientations. For example, an element shown as being integrally formed may be composed of multiple parts or elements, the positions of these elements may be reversed or otherwise changed, and the nature or number or position of discrete elements may be altered or varied. Other substitutions, modifications, changes and omissions may also be made in the design, operating conditions and arrangement of the disclosed elements and operations without departing from the scope of the present disclosure.

[0112] References to element positions herein (e.g., "top," "bottom," "above," "below") are only used to describe the orientation of various elements in the drawings. According to other exemplary embodiments, the orientation of various elements may be different, and such variations are intended to be covered by the present disclosure.

Claims

1. A method comprising: Transmitting, by a device, video data of the device at a first time instance and first coordinates of a first viewport to a server, wherein the video data includes one or more objects in the first viewport; transmitting, by the device, second coordinates of a second viewport of the device at a second time instance; receiving, by the device, data corresponding to one or more objects within the second viewport from the server, the data being received in sequence according to coordinates of the one or more objects relative to the second viewport; as well as Data associated with the one or more objects is rendered, by the device, within a third viewport at a third time instance.

2. The method according to claim 1, comprising: The device receives the data from the server in the order according to the coordinates of the one or more objects relative to the reference point in the second viewport; and preferably, wherein the one or more objects include a first object located at a first distance from the reference point and a second object located at a second distance from the reference point, the method further comprising: When the second distance is greater than the first distance, the device receives the data of the first object before receiving the data of the second object from the server.

3. The method according to claim 1 or 2, comprising: First coordinates of the first viewport are determined, by the device, based on an orientation of the device using sensor data of the device.

4. The method according to any one of the preceding claims, further comprising: establishing, by the device, one or more parameters with the server for at least one of processing or transmitting the video data; as well as The video data is provided by the device in accordance with the established one or more parameters.

5. A method according to any one of the preceding claims, comprising: determining, by the device, whether available computing resources of the device meet a threshold criterion; as well as Preferably, the method further comprises: at a fourth time instance, in response to the available computing resources satisfying the threshold criterion: detecting, by the device, an object from the second video data; determining, by the device, one or more characteristics of the object; and The coordinates of the object and the one or more features are transmitted by the device to the server.

6. A method according to any one of the preceding claims, wherein: The one or more objects include at least one object located within the first viewport and outside the second viewport, the method comprising: determining, by the device, that the at least one object is located within the third viewport; and Data associated with the at least one object is rendered, by the device, within the third viewport.

7. A method according to any one of the preceding claims, wherein: The one or more objects include at least one object located within the second viewport, the method comprising: determining, by the device, that the at least one object is outside the third viewport; and Rendering data corresponding to the at least one object within the third viewport is abandoned by the device.

8. A device comprising: A transceiver, the transceiver being configured to: transmitting, to a server, video data of the device at a first time instance and first coordinates of a first viewport, the video data including one or more objects in the first viewport; transmitting to the server second coordinates of a second viewport of the device at a second time instance; receiving data corresponding to one or more objects within the second viewport from the server, the data being received in sequence according to coordinates of the one or more objects relative to the second viewport; as well as At least one processor is configured to render data associated with the one or more objects within a third viewport at a third time instance.

9. The device according to claim 8, wherein: The transceiver is configured to: receiving the data from the server in the order according to coordinates of the one or more objects relative to a reference point in the second viewport; and Preferably, wherein the one or more objects include a first object located at a first distance from the reference point and a second object located at a second distance from the reference point, wherein the transceiver is further configured to: When the second distance is greater than the first distance, the data of the first object is received before the data of the second object is received from the server.

10. The device according to claim 8 or 9, wherein: The at least one processor is configured to: Using sensor data of the device, first coordinates of the first viewport are determined based on an orientation of the device.

11. The apparatus according to any one of claims 8 to 10, wherein: The at least one processor is configured to: establishing one or more parameters with the server for at least one of processing or transmitting the video data; and The video data is provided according to the established one or more parameters.

12. The apparatus according to any one of claims 8 to 11, wherein: The at least one processor is configured to: At a fourth time instance, determining whether available computing resources of the device meet a threshold criterion; and Preferably, the at least one processor is configured to: In response to the available computing resources satisfying the threshold criterion, detecting an object from the second video data; and determining one or more characteristics of the object; Wherein, the transceiver is configured to transmit the coordinates of the object and the one or more characteristics to the server.

13. The apparatus according to any one of claims 8 to 12, wherein: The one or more objects include at least one object located within the first viewport and outside the second viewport, wherein the at least one processor is configured to: determining that the at least one object is within the third viewport; and Data associated with the at least one object is rendered within the third viewport.

14. The apparatus according to any one of claims 8 to 13, wherein: The one or more objects include at least one object located within the second viewport, wherein the at least one processor is configured to: determining that the at least one object is outside of the third viewport; and Rendering data corresponding to the at least one object within the third viewport is abandoned. 15 . A non-transitory computer-readable medium storing instructions, which, when executed by at least one processor, cause the at least one processor to implement the method according to claim 1 .

Images