Method and system for tracking using pixel sensors of event cameras

Abstract

Various implementations disclosed herein include devices, systems, and methods for obtaining information from a first electronic device (e.g., to improve interaction with a computer-generated reality CGR environment). In some implementations, at a second electronic device having a processor, event camera data corresponding to modulated light (e.g., light including an information signal in its modulation) emitted by a light source on the first electronic device and received at an event camera is obtained. The second electronic device is capable of identifying information from the first electronic device based on a modulation pattern of the modulated light detected according to the event camera data. In some implementations, the second electronic device is the same electronic device having an event camera (e.g., a laptop computer) or a different electronic device that receives event data (e.g., a server that receives event data from a laptop computer).

Description

Technical Field

[0001] The present disclosure as a whole relates to systems, methods and devices for obtaining information from a first electronic device, and specifically for obtaining event camera data corresponding to modulated light (e.g., light including an information signal in its modulation) emitted by a light source on the first electronic device and received at a sensor. Background Art

[0002] Radio communication (e.g., using short wavelength UHF radio waves) for exchanging data between a first electronic device (e.g., an accessory) and a second electronic device (e.g., a host device) over a short distance requires pairing of the two electronic devices. In addition, such radio communication has a relatively low data rate, which can increase communication delays between the two electronic devices. Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for sending information from one electronic device to another electronic device using modulated light, the modulated light being detected using an event camera. Some embodiments involve obtaining event camera data using an event camera at a receiving electronic device, the event camera data indicating the intensity of the received modulated light incident on the event camera. The event camera data is interpreted (e.g., at a receiving electronic device) to identify information sent by a sending first electronic device. In some embodiments, the first electronic device is a laptop computer, a stylus, an input ball, a speaker, a TV device, a display device, a watch, a wearable electronic device, etc. For example, a stylus can modulate the light output from an LED on the tip of the stylus to notify a nearby laptop of the stylus's identity, position, relative position, absolute position in world space, IMU data, movement, posture, or operator action (e.g., button click) at the stylus.

[0004] In some implementations, at a second electronic device having a processor, event camera data corresponding to modulated light emitted by a light source on a first electronic device is obtained. In some implementations, the second electronic device is an electronic device in which processing of the event camera data occurs. In some implementations, the second electronic device is the same electronic device that includes an event camera (e.g., a laptop computer). In some implementations, the second electronic device is a different electronic device that receives event data from an electronic device having an event camera (e.g., a server that receives event data from a laptop computer). The second electronic device identifies information from the first electronic device based on detecting a modulation pattern of the modulated light according to the event camera data.

[0005] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0007] Figure 1 is a block diagram of an example system according to some implementations.

[0008] Figure 2 is a block diagram of an example controller according to some implementations.

[0009] Figure 3 is a block diagram of an exemplary electronic device according to some implementations.

[0010] Figure 4-Figure 6 is a block diagram of various exemplary configurations of an event camera and a modulated light source distributed between a first electronic device, a second electronic device, and a third electronic device according to some specific implementations.

[0011] Figure 7 is a block diagram of a pixel sensor for an event camera and an exemplary circuit diagram of a pixel sensor according to some specific implementations.

[0012] Figure 8 is a flow chart illustrating an example of a method of obtaining information from a first electronic device according to some specific implementations.

[0013] Fig. 9 is a block diagram of an example configuration of multiple accessory devices in communication with a host electronic device according to some implementations.

[0014] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0015] Many details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings illustrate only some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the example implementations described herein. Although Figures 1 to 3 Depicted are exemplary implementations involving electronic devices including, but not limited to, watches and other wearable electronic devices, mobile devices, laptops, desktop computers, HMDs, gaming devices, home automation devices, accessory devices, and other devices that include or use image capture devices.

[0016] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some specific implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary specific implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device (e.g., a laptop computer) 120, one or all of which may be in a physical environment 105. The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0017] In some implementations, the controller 110 is configured to manage and coordinate a user's computer generated reality (CGR) environment. In some implementations, the controller 110 includes a suitable combination of software, firmware, or hardware. Figure 2 Controller 110 is described in further detail. In some implementations, controller 110 is a computing device that is located locally or remotely relative to physical environment 105.

[0018] In one example, the controller 110 is a local server located within the physical environment 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some implementations, the controller 110 is communicatively coupled to the corresponding electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0019] In some implementations, the controller 110 and a corresponding electronic device (eg, 120 ) are configured to together present a CGR environment to a user.

[0020] In some implementations, the electronic device 120 is configured to present a CGR environment to a user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, or hardware. Figure 3 The electronic device 120 is described in more detail. In some implementations, the functionality of the corresponding controller 110 is provided by or combined with the electronic device 120, for example, in the case of an electronic device used as a stand-alone unit.

[0021] According to some specific implementations, when the user appears in the physical environment 105, the electronic device 120 presents a CGR environment to the user. A CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one physical law. For example, the CGR system can detect a person's head rotation, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. In some cases (e.g., for accessibility reasons), the adjustment of features of virtual objects in the CGR environment can be performed in response to a representation of physical movement (e.g., a voice command).

[0022] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0023] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0024] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0025] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0026] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0027] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0028] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0029] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0030] Figure 2is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), Bluetooth, ZIGBEE, or similar type interface), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0031] In some implementations, the one or more communication buses 204 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a trackpad, a joystick, one or more microphones, one or more speakers, one or more image capture devices or other sensors, one or more displays, etc.

[0032] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (CGRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and a computer-generated reality (CGR) experience module 240.

[0033] Operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks.

[0034] In some embodiments, the CGR module 240 is configured to create, edit, present or experience a CGR environment. The CGR module 240 is configured to present virtual content to be used as part of the CGR environment to one or more users. For example, a user may view and otherwise experience a CGR-based user interface that allows the user to select, place, move and otherwise present the CGR environment based on the virtual content position, such as via gestures, voice commands, input device input, etc. In some embodiments, the CGR module 240 includes a modulated light analysis unit 242. The modulated light analysis unit 242 is configured to use modulated light from a sending electronic device, which is detected at a receiving electronic device using an event camera. Although these modules and units are shown as residing on a single device (e.g., electronic device 120), it should be understood that in other embodiments, any combination of these modules and units may be located in a separate computing device.

[0035] Although these modules and units are shown as residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of these modules and units may be located in separate computing devices. Figure 2 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0036] Figure 3is a block diagram of an example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more internal or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0037] In some implementations, one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0038] In some implementations, one or more displays 312 are configured to present a CGR environment to a user. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), or similar display types. In some implementations, one or more displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, an electronic device may include a single display. For another example, an electronic device may include a display for each eye of a user.

[0039] The memory 320 includes a high-speed random access memory, such as DRAM, CGRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or their subsets, including an optional operating system 330 and a CGR module 340.

[0040] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks.

[0041] In some embodiments, the CGR module 340 is configured to create, edit, present or experience a CGR environment. The CGR module 340 is configured to present virtual content to be used as part of the CGR environment to one or more users. For example, a user may view and otherwise experience a CGR-based user interface that allows the user to select, place, move and otherwise present the CGR environment based on the virtual content position, such as via gestures, voice commands, input device input, etc. In some embodiments, the CGR module 340 includes a modulated light analysis unit 342. The modulated light analysis unit 342 is configured to use modulated light from a sending electronic device that is detected at a receiving electronic device using an event camera. Although these modules and units are shown as residing on a single device (e.g., electronic device 120), it should be understood that in other embodiments, any combination of these modules and units may be located in a separate computing device.

[0042] also, Figure 3 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0043] Figure 4-Figure 6The system depicted in provides a block diagram of various exemplary configurations of how an event camera and a modulated light source may be distributed among a first electronic device, a second electronic device, and a third electronic device according to some specific implementations. Figure 4 is a block diagram of an exemplary system including a first electronic device communicatively coupled to a second electronic device. Figure 4 As shown, exemplary system 400 includes a first electronic device 410 communicatively coupled to a second electronic device 420 in a physical environment 405. Figure 4 In the configuration of, the second electronic device 420 includes an event camera 424, and the first electronic device 410 includes one or more modulated light sources 412. In some specific implementations, the modulated light of the modulated light source 412 includes a modulated amplitude (e.g., polarization), phase, or frequency of the radiated light. In some specific implementations, the event camera 424 is always turned on due to its low power consumption. In some specific implementations, when detecting the modulated light source 412, the event camera 424 runs at a very high frame rate. In some specific implementations, the event camera 424 detects the modulated light faster than 1000htz. Therefore, when the modulated light from the modulated light source 412 is detected, the event camera 424 can reduce the delay in the communication. In addition, when the event camera 424 is used to detect the modulated light from the modulated light source 412, the second electronic device 420 can smoothly track the first electronic device 410.

[0044] like Figure 4 As shown, the second electronic device 420 also includes a processor 426 configured to use event camera data output by the event camera 424. In some specific implementations, the event camera data is a stream of pixel events output by the event camera 424, which is caused by the light received from the modulated light source 412. In some specific implementations, when light from the modulated light source 412 is detected, the processor 426 can use the event camera data output by the event camera 424 to obtain information about the first electronic device 410 or from the first electronic device 410. When obtaining information from the first electronic device 410, the second electronic device 420 can use the event camera data corresponding to the information signal encoded by the received modulated light. In addition, the processor 426 can use the detection of modulated light from the modulated light source 412 to determine information about the first electronic device 410, such as the relative position (e.g., spatial orientation or movement) of the first electronic device 410 in the physical environment 405 or relative to the second electronic device 420.

[0045] In some implementations, the first electronic device 410 may be an electronic accessory device capable of encoding information into modulated light output by a modulated light source. In some implementations, the first electronic device 410 is a laptop computer, a stylus, an input device, a smart speaker, an HMD, a smart TV, a smart watch, a wearable electronic device, an accessory, a display device, etc. Figure 4 As shown, in some implementations, the first electronic device 410 can be a stylus with a modulated light source 412 at one end. In some implementations, the first electronic device 410 can include more than one modulated light source 412, and the event camera 424 can detect multiple concurrent encoded information from different modulated light sources. In some implementations, there can be multiple different first electronic devices that use different modulations in corresponding modulated light sources, and the event camera 424 detects multiple concurrent different encoded information from each of the multiple different first electronic devices.

[0046] In some implementations, the modulated light source 412 may be a non-directional light source. In some implementations, the modulated light source 412 is an LED or the like. In some implementations, the modulated light source 412 operates within a first wavelength range. For example, the modulated light source 412 may be an infrared (IR) light source. For example, the modulated light source 412 may be a near infrared (NIR) light source. For example, the modulated light source 412 may be a light source that emits within a visible wavelength range. In various implementations, the field of view (FOV) of the event camera 424 and the emission cone of the modulated light source 412 overlap for the event camera 424 to detect the modulated light source 412. In some implementations, the event camera 424 is tuned or adapted to detect light in a first wavelength range (e.g., IR). In some implementations, the event camera 424 detects light in a wavelength range greater than the first wavelength range and includes the first wavelength range.

[0047] In some implementations, the first electronic device 410 may be an accessory of different types. In one implementation, the accessory first electronic device 410 is a 3D input device having a predetermined shape (e.g., a manipulable cube, a rounded sphere; an egg-shaped ovoid, etc.) that is illuminated with modulated light for use in a CGR environment visible at the second electronic device 420. In this exemplary implementation, an operator action (e.g., a button push, horizontal or vertical movement, twisting, partial rotation, etc.) at the first electronic device 410 as an input device illuminated with modulated light demonstrates the user action in the CGR environment at the second electronic device 420. In addition, in this exemplary implementation, when the input first electronic device 410 has a predetermined illuminated shape (e.g., a size or array of modulated light sources), the second electronic device 420 can use the spatial position of the received modulated light to determine the absolute distance or rotation of the first electronic device 410. In some implementations, modulation with a known pattern can be used as a robust different feature of the modulated light source 412 that is detected and tracked. Additionally, in this exemplary implementation, when the input first electronic device 410 has a predetermined illuminated shape, the second electronic device 420 can determine the absolute distance or rotation of the first electronic device using the spatial position of the received light in the FOV of the event camera 424, regardless of whether the light is modulated. In some implementations, the shared CGR environment visible at the second electronic device 420 can enable multiple first electronic devices 410 in the shared CGR environment to interact simultaneously.

[0048] In various implementations, the modulated light from the modulated light source 412 itself generates events at the event camera 424 (e.g., such events typically have a frequency that is the same as the pulse frequency of the modulated light). However, the non-modulated light is effectively "invisible" to the event camera 424, as an event is initiated at the event camera 424 only when motion occurs in a portion of the physical environment 405 illuminated by the non-modulated light (e.g., motion that includes the non-modulated light).

[0049] Figure 5 is a block diagram of an exemplary system including a first electronic device, a second electronic device, and a third electronic device communicatively coupled together. Figure 5 As shown, exemplary system 500 includes a first electronic device 410, a second electronic device 420, and a third electronic device 430 communicatively coupled together in a physical environment 505. Figure 5 As shown, the first electronic device 410 includes a modulated light source 412 and an inertial measurement unit ("IMU") 417, and the second electronic device 420 includes a processor 426. Figure 5As shown, the third electronic device 430 includes an event camera 434. The IMU is a computing device that provides inertial data by sampling signals generated by one or more sensors of the IMU. One or more sensors may include one to three axis accelerometers, one to three axis axis gyroscopes, one to three axis magnetometers, barometric pressure sensors, and the like. In some implementations, the processor 426 may use the event camera data output by the event camera 434 to obtain information from the first electronic device 410. For example, the IMU data from the IMU 417 (e.g., the spatial orientation or movement of the first electronic device 410) may be used to encode and transmit the modulated light of the modulated light source 412 detected by the free event camera 434. In some implementations, the transmission or detection of the embedded IMU data is used to track or maintain the state of the first electronic device 410, including XYZ and roll, pitch, yaw (e.g., 6 degrees of freedom), which may be maintained at the second electronic device 420. In addition, the processor 426 can again use detection of the modulated light itself from the modulated light source 412 to determine the relative position (e.g., spatial orientation or movement) of the first electronic device 410 in the physical environment 505 or relative to the third electronic device 430 (e.g., event camera 424).

[0050] Figure 6 is a block diagram of an exemplary system including a first electronic device and a second electronic device. Figure 6 As shown, exemplary system 600 includes a first electronic device 610 and a second electronic device 620 communicatively coupled together in a physical environment 605. Figure 6 As shown, the modulated light source 412 , the event camera 414 , the processor 416 , and the IMU 417 are disposed on the first electronic device 610 , and the modulated light source 422 , the event camera 424 , the processor 426 , and the IMU 427 are disposed on the second electronic device 620 .

[0051] like Figure 6 As shown, the second electronic device 620 in the system 600 may include various capabilities or functions of each of the first electronic device 410 and the second electronic device 420 in the system 400, or various capabilities or functions of each of the first electronic device 410, the second electronic device 420, and the third electronic device 430 in the system 500. Similarly, the first electronic device 610 in the system 600 includes various capabilities or functions of each of the first electronic device 410 and the second electronic device 420 in the system 400, or various capabilities or functions of each of the first electronic device 410, the second electronic device 420, and the third electronic device 430 in the system 500. In some specific implementations, the first electronic device 610 and the second electronic device 620 include other communication channels (e.g., radio waves).

[0052] Figure 7 is a block diagram of a pixel sensor for an exemplary event camera and an exemplary circuit diagram of a pixel sensor according to some specific implementations. Figure 7 As shown, the pixel sensors 715 can be arranged in a 2D matrix 710 of rows and columns relative to the electronic device (e.g., Figure 1 The electronic device 120 is set on the event camera at a known position. Figure 7 In the example of , each pixel sensor in pixel sensors 715 is associated with an address identifier defined by a row value and a column value.

[0053] Figure 7 Also shown is an example circuit diagram of a circuit 720 suitable for implementing pixel sensor 715. Figure 7 In the example of FIG. 7 , circuit 720 includes a photodiode 721, a resistor 723, a capacitor 725, a capacitor 727, a switch 729, a comparator 731, and an event compiler 732. In operation, a voltage is generated on the photodiode 721, which is proportional to the intensity of light incident on the pixel sensor. The capacitor 725 is parallel to the photodiode 721, and therefore the voltage on the capacitor 725 is the same as the voltage on the photodiode 721.

[0054] In circuit 720, switch 729 is interposed between capacitor 725 and capacitor 727. Thus, when switch 729 is in a closed position, the voltage across capacitor 727 is the same as the voltage across capacitor 725 and photodiode 721. When switch 729 is in an open position, the voltage across capacitor 727 is fixed to the previous voltage across capacitor 727 when switch 729 was last in a closed position. Comparator 731 receives and compares the voltages across capacitor 725 and capacitor 727 on the input side. If the difference between the voltage across capacitor 725 and the voltage across capacitor 727 exceeds a threshold amount ("comparator threshold"), an electrical response (e.g., voltage) indicative of the intensity of light incident on the pixel sensor is present on the output side of comparator 731. Otherwise, there is no electrical response on the output side of comparator 731.

[0055] When the electrical response is present on the output side of the comparator 731, the switch 729 is turned to the closed position, and the event compiler 732 receives the electrical response. Upon receiving the electrical response, the event compiler 732 generates a pixel event and adds the pixel event with information indicating the electrical response (e.g., the value or polarity of the electrical response). In one specific implementation, the event compiler 732 also adds the pixel event with one or more of the following: timestamp information corresponding to the time point at which the pixel event was generated and an address identifier corresponding to the specific pixel sensor that generated the pixel event.

[0056] The event camera typically includes a plurality of pixel sensors, such as pixel sensor 715, each of which outputs a pixel event in response to detecting a light intensity change exceeding a comparison threshold. When aggregated, the pixel events output by the plurality of pixel sensors form a pixel event stream output by the event camera. In some specific implementations, the light intensity data obtained from the pixel event stream output by the event camera is used to determine the correspondence of the changes between two electronic devices (e.g., the first electronic device 410 and the second electronic device 420). When the event camera is set on one of the first electronic device and the second electronic device (e.g., the first electronic device 610 and the second electronic device 620), at least a portion of the light intensity change corresponds to light emitted by a plurality of light sources set on another device among the first electronic device and the second electronic device.

[0057] Figure 8 is a flowchart illustrating an exemplary method of obtaining information from a first electronic device according to some specific implementations. In some specific implementations, the electronic device (e.g., Figures 1 to 3 The method 800 is performed by a controller 110 or an electronic device 120 of the embodiment of the present invention. The method 800 may be performed at a mobile device, an HMD, a desktop computer, a laptop computer, a server device, or by multiple devices communicating with each other. In some specific implementations, the method 800 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 800 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0058] In some implementations, method 800 is used to obtain information from a first electronic device. In some implementations, the first electronic device is a laptop computer, a stylus, an input sphere, a smart speaker, a smart TV, a smart watch, a wearable electronic device, etc. In various implementations, method 800 operates at a second electronic device having a processor. In some implementations, the second electronic device includes an event camera (e.g., a laptop computer with an event camera). In some implementations, the second electronic device is a different electronic device that receives event data from an electronic device including an event camera (e.g., a server that receives event data from a laptop computer with an event camera).

[0059] At block 810, method 800 obtains event camera data corresponding to modulated light emitted by a light source at a first electronic device (e.g., light including an information signal in the modulated signal). In some implementations, the light source is one or more LEDs on the first electronic device. In some implementations, the generated event camera data is based on light intensity changes detected at a pixel sensor of the event camera (e.g., the event camera data includes events triggered by light intensity changes in a pixel sensor of the event camera that exceed a comparator threshold).

[0060] At box 820, the method 800 identifies information from the first electronic device based on detecting a modulation pattern of the modulated light according to the event camera data. In some specific implementations, the modulated light includes a modulated amplitude (e.g., modulated polarization), a modulated phase, a modulated frequency, etc. In some specific implementations, exemplary information identified from the first electronic device includes, but is not limited to, the identity of the first electronic device, the relative position of the first electronic device, the absolute position of the first electronic device in a 3D spatial coordinate system (e.g., world space or physical environment 405), IMU data of the first electronic device, movement of the first electronic device, posture of the first electronic device, (one or more) button clicks occurring at the first electronic device, operator actions occurring at the first electronic device, etc.

[0061] In some implementations, at block 810, modulated light emitted by one or more light sources at a first electronic device is received at an event camera located at a second electronic device. In some implementations, a third electronic device separate from the second electronic device includes the event camera. In some implementations, at block 810, modulated light emitted by one or more light sources at a first electronic device is received at the third electronic device and the modulated light is transmitted to the second electronic device. In some implementations, the second electronic device is where processing is performed to implement method 800.

[0062] In some implementations, at block 820, method 800 determines a position of the first electronic device relative to a physical environment or a second electronic device based on information determined from modulated light from the first electronic device located in the FOV of the event camera. In some implementations, at block 820, method 800 identifies movement of the first electronic device relative to the physical environment or the second electronic device. In some implementations, the event camera at the second device is a spatial sensor (e.g., the event camera can locate an object (e.g., movement) by the object's x, y coordinates on an image sensor plane).

[0063] Fig. 9 is a block diagram of an exemplary configuration of multiple accessory devices in communication with a host electronic device according to some implementations. Fig. 9As shown, accessory device 910a using modulated light source 912a and accessory device 910b using modulated light source 912b are detected by event camera 924 at host electronic device 920 to provide electronic device positioning and identification, as well as point communications (e.g., IMU data and operator actions). Fig. 9 As shown, the same communication capability can be implemented bidirectionally by the host electronic device 920 using modulated light sources 922, which are detected by event cameras 914a, 914b at the accessory devices 910a, 910b, respectively. In some specific implementations, the modulated light sources 912a, 912b use different modulations (e.g., different frequencies or encodings) for simultaneous communication with the host electronic device 920.

[0064] In one implementation, the first electronic devices 910a, 910b may be smart speakers (or sound bars) that are differently positioned in a room as the physical environment 905. In this example, when a second electronic device 920 (e.g., a host laptop, tablet) enters the room physical environment 905, the accessory speakers 910a, 910b are detected by the host second electronic device 920 using modulated optical communication, which allows the accessory speakers 910a, 910b to direct their audio emissions to provide stereo sound (e.g., surround sound, etc.) for the current location of the host second electronic device 920. In this exemplary implementation, the modulated optical communication may be intermittent or periodic (e.g., discontinuous) and still provide adequate audio control.

[0065] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0066] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0067] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0068] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the blocks presented in the above examples can be changed, for example, the blocks can be reordered, combined and / or divided into sub-blocks. Some blocks or processes can be executed in parallel.

[0069] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0070] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0071] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0072] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0073] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method for obtaining information from a first electronic device, the method comprising: At a second electronic device having a processor: obtaining event camera data corresponding to information encoded within modulated light emitted by a light source on the first electronic device and received at an event camera, the event camera data being generated based on light intensity variations detected at pixel sensors of the event camera, wherein the event camera generates pixel events, each respective pixel event being generated in response to a particular pixel sensor within a pixel array of the event camera detecting a light intensity variation exceeding a comparison threshold; as well as The information from the first electronic device is identified based on detecting a modulation pattern of the modulated light according to the event camera data, wherein a position, movement, or operator action of the first electronic device is determined based on the information detected in the modulation pattern of the modulated light.

2. The method according to claim 1, wherein the position of the first electronic device is determined based on the position of an event camera event in the field of view FOV of the event camera and the information detected in the modulation pattern of the modulated light. The method of claim 1 , wherein the second electronic device comprises the event camera. The method according to claim 1 , wherein a third electronic device includes the event camera, the third electronic device being separate from the second electronic device. The method of claim 1 , wherein the modulated light comprises an information signal containing the information. The method of claim 1 , wherein the modulated light comprises modulated amplitude, phase or frequency of a radiated wave.

7. The method of claim 1, wherein the event camera data corresponds to a pixel event triggered based on a change in light intensity at a pixel sensor exceeding a comparator threshold. 8 . The method of claim 1 , wherein the information from the first electronic device comprises a unique identifier of the first electronic device or a device type of the first electronic device.

9. The method of claim 1, further comprising determining a position of the first electronic device relative to the second electronic device based on the information from the first electronic device.

10. The method of claim 1, further comprising determining a position of the first electronic device relative to a physical environment based on the information from the first electronic device.

11. The method of claim 1, wherein the information from the first electronic device identifies movement of the first electronic device.

12. The method of claim 1, wherein the information from the first electronic device identifies input received at the first electronic device.

13. The method of claim 1, wherein the information from the first electronic device provides device identification and location of the first electronic device.

14. The method of claim 1, wherein the event camera operates at a frame rate greater than or equal to 1000 frames per second.

15. The method of claim 1, wherein the event camera enables high-speed tracking of the first electronic device.

16. The method of claim 1, wherein the first electronic device is configured to function as an input device to the second electronic device.

17. A system for obtaining information from a first electronic device, comprising: a non-transitory computer-readable storage medium; as well as one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: obtaining event camera data corresponding to information encoded within modulated light emitted by a light source on a first electronic device and received at an event camera, the event camera data being generated based on light intensity variations detected at pixel sensors of the event camera, wherein the event camera generates pixel events, each respective pixel event being generated in response to a particular pixel sensor within a pixel array of the event camera detecting a light intensity variation exceeding a comparison threshold; as well as The information from the first electronic device is identified based on detecting a modulation pattern of the modulated light according to the event camera data, wherein a position, movement, or operator action of the first electronic device is determined based on the information detected in the modulation pattern of the modulated light.

18. A non-transitory computer-readable storage medium storing computer-executable program instructions on a computer to perform operations including: At a second electronic device having a processor: obtaining event camera data corresponding to information encoded within modulated light emitted by a light source on the first electronic device and received at an event camera, the event camera data being generated based on light intensity variations detected at pixel sensors of the event camera, wherein the event camera generates pixel events, each respective pixel event being generated in response to a particular pixel sensor within a pixel array of the event camera detecting a light intensity variation exceeding a comparison threshold; and The information from the first electronic device is identified based on detecting a modulation pattern of the modulated light according to the event camera data, wherein a position, movement, or operator action of the first electronic device is determined based on the information detected in the modulation pattern of the modulated light.

Images