Remote camera control by gaze detection

Abstract

An apparatus and system for camera control of a remote camera. The apparatus may include a display device. The apparatus may also include a first camera device mounted to the display device. The first camera device may capture an image of an iris of a user of the display device. The apparatus may include processing circuitry coupled to the first camera device. The processing circuitry may detect a gaze direction of the iris based on the image. The processing circuitry may encode a control signal based on the gaze direction to control a position of a second camera device remote from the first camera device.

Description

Technical Field

[0001] The embodiments described herein generally relate to a camera device system for video conferencing, and in embodiments (but not by way of limitation) relate to remote camera device control during video conferencing. Background Art

[0002] During a video conference call, remote participants may wish to control the camera device at the main video conferencing location. Some systems may rely on audio input or other criteria to zoom in or focus on the current speaker or other elements in the video conference. However, remote participants may prefer to zoom in on other areas of the remote conference room, or on documents or other visual aids in the conference room. Brief Description of the Drawings

[0003] In the drawings, which are not necessarily to scale, like reference numerals may describe like components in different figures. Like reference numerals with different letter suffixes may represent different instances of similar components. Some embodiments are shown by way of example and not limitation in the figures of the accompanying drawings.

[0004] Figure 1 A device for pan and tilt control of a remote camera device based on gaze detection according to some embodiments is shown.

[0005] Figure 2 A system for pan and tilt control of a remote camera device based on gaze detection according to some embodiments is shown.

[0006] Figure 3 The type of camera device used in a system for pan and tilt control of a remote camera device based on gaze detection according to some embodiments is shown.

[0007] Figure 4 is a flowchart of a method for pan and tilt control of a remote camera device based on gaze detection according to some embodiments.

[0008] Figure 5 A detailed overview of example components within a computing device in a computing system is provided. Detailed Description of Embodiments

[0009] Due to the decline in the cost of meeting equipment and the popularity of high-speed Internet, video conferencing has recently become increasingly popular. Generally, business organizations will designate physical spaces specifically for video conferencing. Conference attendees can join the conference locally or remotely. There may be multiple different speakers present locally or remotely, and other visual aids such as whiteboards, prototype products, etc. may be used.

[0010] The designed physical space may include at least one camera device, which is positioned to capture a video stream of all or a part of a specified meeting environment (including local participants, visual aids, etc.). Some systems can detect activities within the physical space and accordingly adjust the image boundaries (field of view) displayed to remote meeting participants. For example, some systems can detect people or objects within a larger physical space and reconstruct (or "zoom") the captured video stream around these people / objects. Generally, once the camera device has been configured or zoomed, no further camera device control is performed. For example, one or more camera devices in use may remain focused on the front of the specified meeting environment.

[0011] However, remote participants may wish to zoom in on different people or scenes at different times throughout the video conference. Users may also be interested in specific details of the video conference. For example, remote participants may wish to see the facial expressions of different people in the room or focus on a prototype product being demonstrated. There are methods for automatically evaluating the salience of scene content and zooming / cropping, but this may not represent the desired viewing area of remote participants at a remote location.

[0012] Methods and devices according to embodiments of the present disclosure solve these problems by combining gaze detection (which determines the observer's perspective and regions of interest in the image) with control of remote camera devices (e.g., one or more camera devices present in a remote meeting environment) based on the observer's gaze. The methods and devices can allow the observer to view the content of interest at a high resolution or based on other customized requirements or preferences of the observer.

[0013] Figure 1 Device 100 for pan and tilt control of a remote camera device based on gaze detection according to some embodiments is shown. As Figure 1 shown, remote user 102 may be participating in a video conference using display device 108. Display device 108 displays a user interface including one or more interface elements to user 102. In the example, the user interface includes video conference elements, which include presenter 104 speaking beside whiteboard 106. User's computer 114 may include camera device 116 installed thereon. One or more other camera devices 116 may be included elsewhere in display device 108.

[0014] The user's iris 110 can focus on various positions on the display device 108. For example, the iris 110 of a remote user can focus on point 112. The remote user 102 may be looking at point 112 to detect the facial expression of the person at point 112, for example, or because point 112 contains some other point of interest. The processing circuitry 120 can use an image of the user 102's iris 110 captured by one or more imaging devices 116 to determine the gaze direction of the remote user 102. For example, based on the known coordinates of the elements on the display device 108, the processing circuitry 120 can determine which elements the user 102 is focusing on by electronically analyzing the positions of the user elements within the user interface related to the gaze position of the user 102 using the detected gaze of the user 102.

[0015] As previously mentioned herein, some available systems can zoom in on the presenter 104 or crop a portion of the video to focus on the presenter 104 or other predetermined positions. However, the method according to the embodiments provides the following method: performing gaze detection based on the gaze direction of the remote user 102 to direct the remote imaging device system to focus on the content related to the remote user 102. For example, the processing circuitry 120 can encode a control signal based on the gaze direction to control the position of a second imaging device away from the first imaging device, as described with respect to Figure 2 and Figure 3 described.

[0016] Figure 2 A system 200 for pan and tilt control based on a remote imaging device through gaze detection according to some embodiments is shown. The processing circuitry 120 ( Figure 1 ) can encode a control signal 204 to be transmitted to control the position of a second imaging device 206 away from the first imaging device 116 based on the detected gaze direction 202. The control signal 204 can include metadata or other data appended to the signal for controlling the imaging device 206. The control signal 204 can include data or control signals for controlling at least one of the pan function, tilt function, and crop function of the second imaging device 206.

[0017] The second imaging device 206 can be used to provide a focus 208 at a remote location to match or align with the gaze direction 202 of the user 102. For example, if the user 102 looks down as shown by the gaze direction 202, then the imaging device 206 can be controlled to focus down as shown by the focus 208. In the illustrated example, the user 102 may be focusing on a person 210 at a remote location 212 (e.g., a remote video conference room).

[0018] More than one imaging device can be provided at the remote location 212. Figure 3Illustrates a type of camera device used in a system for remote camera device pan and tilt control based on gaze detection according to some embodiments. For example, a wide-angle (or wide field of view (WFoV) camera device) 300 can be provided at a remote location 212. The wide-angle camera device 300 can be used to crop the region of interest 302. Although the wide-angle camera device 300 may be useful in some cases, as seen at view 304, it may result in blurring. In some examples, a telephoto lens or other high-resolution camera device 306 can be provided to display high-resolution details of the region of interest 308. Using a higher-resolution (e.g., telephoto) camera device 306, focusing on the region of interest 308 can be achieved with reduced blurring. The processing circuitry 120( Figure 1 ) can encode the second control signal (or provide additional data in the first control signal) to control the position of two or each of the camera devices 300, 306.

[0019] Referring again to Figure 1 , a group of remote users 102 may be viewing a video conference, each remote user on a separate display ( Figure 1 not shown) or each remote user on the same display device 108. The corresponding processing circuitry 120 of each user device can access a corresponding one or more camera devices 116 to detect the gaze directions of two or more of these users 102. In some embodiments, the processing circuitry 120 can determine the priority levels of multiple users and encode the control signal based on the highest-priority group of users. In other embodiments, other methods can be used to control the remote camera device, such as by polling or surveying. In an example, the processing circuitry 120 can encode the control signal based on a determination of the user preferences of the largest number of users (or higher-ranking individuals, or other criteria).

[0020] Figure 4 is a flowchart of a method 400 for remote camera device pan and tilt control based on gaze detection according to some embodiments. The method 400 can be implemented by the processing circuitry 120( Figure 1 ), one or more camera devices 116( Figure 1 ) or Figure 1 or Figure 5 other components shown in

[0021] The method 400 can start at operation 402, where the camera device 116 captures an image of the iris 110 of the user 102 of the display device 108.

[0022] Method 400 can proceed to operation 404, where processing circuitry 120 detects the gaze direction of iris 110 based on the image captured in operation 402.

[0023] Method 400 can proceed to operation 406, where processing circuitry 120 encodes a control signal based on the gaze direction to control the position of a camera device away from the display device (e.g., camera device 206( Figure 2 ), or camera device 300 or camera device 306( Figure 3 ). The control signal can include data for controlling at least one of a pan function, a tilt function, and a crop function of the camera device. Multiple control signals can be encoded to control multiple remote camera devices. Alternatively, a single control signal can be encoded with multiple data portions to control multiple remote camera devices.

[0024] Figure 5 A detailed overview of example components within computing device 550 in a computing system is provided. Computing device 550 can be embodied as a type of device, appliance, computer, or other "thing" capable of communicating with other edge, networking, or terminal components. For example, computing device 550 can be embodied as a personal computer, server, smart phone, mobile computing device, smart appliance, in-vehicle computing system (e.g., a navigation system), a stand-alone device having a housing, enclosure, etc., or other device or system capable of performing the described functions. Computing device 550 can perform operations for video conferencing and can be the same as or similar to computer 114( Figure 1 ). In an example, computing device 550 (or its components, such as one or more camera devices and communication circuitry) can be a device disposed at a remote video conferencing location.

[0025] The computing device 550 can include any combination of the hardware components or logic components mentioned herein, and it can include any device usable in an edge communication network or a combination of such networks or be coupled to any device usable in an edge communication network or a combination of such networks. These components can be implemented as integrated circuits (ICs), portions of integrated circuits, discrete electronic devices, or other modules, instruction sets, programmable logic or algorithms, hardware, hardware accelerators, software, firmware, or combinations thereof, which are adapted for edge computing device 550 or as components otherwise included within the enclosure of a larger system.

[0026] Instructions 582 on processor 552 (either alone or in combination with instructions 582 of machine-readable medium 560) may configure the execution or operation of a video conferencing system, device, or method. For example, video conferencing may be performed on a display of device 550, and a camera device or other input device may be used to detect a user's gaze direction related to the display coordinates of video conferencing elements.

[0027] The computing device 550 described above may include other components for performing operations according to example embodiments. The computing device 550 may include processing circuitry in the form of a processor 552, which may be a microprocessor, multi-core processor, multi-threaded processor, ultra-low voltage processor, embedded processor, xPU / DPU / IPU / NPU, special-purpose processing unit, dedicated processing unit, or other known processing element.

[0028] Processor 552 may communicate with system memory 554 via an interconnect 556 (e.g., a bus). Any number of memory devices may be used to provide a given amount of system memory. To provide persistent storage of information such as data, applications, operating systems, etc., a storage device 558 may also be coupled to processor 552 via interconnect 556. Components may communicate via interconnect 556. The interconnect 556 may include any number of technologies, including Industry Standard Architecture (ISA), Extended ISA (EISA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCIx), PCI Express (PCIe), or any number of other technologies.

[0029] The interconnect 556 may couple processor 552 to a transceiver 566 for communication with a connected edge device 562 or with a camera device or computer equipment at a remote video conferencing location as previously described herein. The transceiver 566 may use any number of frequencies and protocols. The wireless network transceiver 566 (or transceivers) may communicate using multiple standards or radios for communication over different ranges. For example, the computing device 550 may communicate with nearby devices (e.g., within approximately 10 meters) using a local transceiver based on Bluetooth Low Energy (BLE) or another low-power radio to save power. It may reach further connected edge devices 562 via or other medium-power radios, e.g., within approximately 50 meters.

[0030] It may include a wireless network transceiver 566 (e.g., a radio transceiver) to communicate with devices or services in the cloud (e.g., edge cloud 595) via a local area network protocol or a wide area network protocol. Given the diversity of the types of applicable communications from the device to another component or network, the applicable communication circuitry used by the device may include any one or more of components 564, 566, 568, or 570 or be embodied by any one or more of components 564, 566, 568, or 570. Thus, in various examples, the applicable means for communication (e.g., receiving, transmitting, etc.) may be embodied by such communication circuitry.

[0031] The sensor 572 may include a camera device and other imaging systems for detecting gaze and for remote locations as described above. The actuator may include motor control or other control to move, tilt, pan, or perform other motions on the camera device or other sensors. The output device 584 may include a display (e.g., for requesting user input or displaying other information related to the applications described herein), and the input device 586 may include a keyboard, a mouse, etc. for receiving user input.

[0032] In an example, the instructions 582 provided by the memory 554, the storage device 558, or the processor 552 may be embodied as a non-transitory machine-readable medium 560 that includes code for instructing the processor 552 to perform electronic operations in the computing device 550. The processor 552 may access the non-transitory machine-readable medium 560 via the interconnect 556. For example, the non-transitory machine-readable medium 560 may be embodied by the device described for the storage device 558, or may include a specific storage unit, such as a storage device and / or a storage disk, which includes optical discs (e.g., digital versatile discs (DVDs), compact discs (CDs), CD-ROMs, Blu-ray discs), flash drives, floppy disks, hard disk drives (e.g., SSDs), or any number of other hardware devices that store information for any duration (e.g., for an extended period, permanently, briefly, temporarily buffered, and / or cached). The non-transitory machine-readable medium 560 may include instructions for instructing the processor 552 to perform a specific sequence or flow of actions, e.g., as described in the flowcharts and block diagrams regarding the operations and functions above. As used herein, the terms "machine-readable medium" and "computer-readable medium" are interchangeable. As used herein, the term "non-transitory computer-readable medium" is explicitly defined to include any type of computer-readable storage device and / or storage disk, but does not include propagated signals and does not include transmission media.

[0033] Although Figure 5Illustrative examples include example components for computing nodes and computing devices respectively, but the examples disclosed herein are not limited thereto. As used herein, "computer" may include some or all of the components in the example components of Figure 5 in different types of computing environments. Example computing environments include edge computing devices (e.g., edge computers) in a distributed networking arrangement such that a particular edge computing device among the participating edge computing devices is a heterogeneous or homogeneous device. As used herein, "computer" may include personal computers, servers, user equipment, accelerators, etc. and any combination thereof. In some examples, distributed networking and / or distributed computing includes any number of such edge computing devices as Figure 5 shown, where each edge computing device may include different sub-components, different memory capacities, I / O capabilities, etc. For example, since some implementations of distributed networking and / or distributed computing are associated with specific desired functions, the examples disclosed herein include Figure 5 different combinations of the components shown to meet the functional objectives of distributed computing tasks. In some examples, one or more of the target functions of a distributed computing task rely on one or more alternative devices / structures located in different parts of the edge networking environment, such as a device for housing a data storage device.

[0034] In this document, as is common in patent documents, the terms "a" or "an" are used to include one or more than one, independent of any other instance or use of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to refer to a non-exclusive or, such that "A or B" includes "A but not B", "B but not A", and "A and B". In the appended claims, the terms "including" and "in which" are used as the ordinary English equivalents of the corresponding terms "comprising" and "wherein". Additionally, in the appended claims, the terms "including" and "comprising" are open-ended, that is, a system, device, article, or process that includes elements is still considered to fall within the scope of that claim, in addition to those listed after such terms in the claim. Further, in the appended claims, the terms "first", "second", "third", etc. are used only as labels and are not intended to indicate a numerical order of their objects.

[0035] The foregoing description is intended to be illustrative and not restrictive. For example, the above-described examples (or one or more aspects of the examples) may be used in combination with other examples. For instance, other embodiments may be utilized by those of ordinary skill in the art upon reading the foregoing description. The abstract is provided to enable the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that the abstract will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing detailed description, various features may be combined to simplify the disclosure. However, the claims may not recite every feature disclosed herein, as an embodiment may be characterized by a subset of such features. Moreover, an embodiment may include fewer features than those disclosed in a particular example. Accordingly, the appended claims are hereby incorporated into the detailed description, where each claim stands on its own as a separate embodiment. The scope of the embodiments disclosed herein should be determined with reference to the appended claims and the full scope of equivalents to such claims.

Claims

1. An apparatus for controlling a camera device, the apparatus comprising: a display device; a first camera device mounted to the display device, the first camera device being configured to capture an image of the iris of a user of the display device; and a processing circuitry coupled to the first camera device and configured to: detect a gaze direction of the iris based on the image; and encode a control signal based on the gaze direction to control a position of a second camera device remote from the first camera device.

2. The apparatus according to claim 1, wherein the control signal includes data for controlling at least one of a pan function, a tilt function, and a crop function of the second camera device.

3. The apparatus according to claim 1, wherein the processing circuitry is configured to: encode a second control signal to control a position of a third camera device of a different type than the second camera device.

4. The apparatus according to claim 3, wherein the third camera device and the second camera device include a wide field of view (WFoV) camera device and a telephoto camera device.

5. The apparatus according to claim 1, wherein the processing circuitry is coupled to a plurality of first camera devices, and wherein the processing circuitry is configured to detect gaze directions of a plurality of users.

6. The apparatus according to claim 5, wherein the processing circuitry is configured to: determine priority levels of the plurality of users, and encode the control signal based on a highest priority user group.

7. The apparatus according to claim 5, wherein the processing circuitry is configured to: provide a request for user input regarding control of the second camera device.

8. The apparatus according to claim 7, wherein the processing circuitry is configured to: encode the control signal based on a determination of user preferences of a user with the highest number.

9. A system for controlling a camera device, the system comprising: a display device; a first camera device mounted to the display device, the first camera device being configured to capture an image of the iris of a user of the display device; a processing circuitry coupled to the first camera device, the processing circuitry being configured to: detect a gaze direction of the iris based on the image; and a communication interface between the processing circuitry and a second camera device, the communication interface being configured to: provide a control signal from the processing circuitry to the second camera device, the control signal being encoded based on the gaze direction.

10. The system according to claim 9, wherein the second camera device includes a wide field of view (WFoV) lens.

11. The system according to claim 9, further comprising a third camera device having a telephoto lens.

12. The system according to claim 9, wherein the control signal includes data for controlling at least one of a pan function, a tilt function, and a crop function of the second camera device.

13. The system according to claim 9, wherein The processing circuit system is coupled to a plurality of first imaging devices, and wherein the processing circuit system is configured to detect the gaze directions of a plurality of users.

14. The system according to claim 13, wherein, the processing circuit system is configured to: determine the priority levels of the plurality of users, and encode the control signal based on the user group with the highest priority.

15. The system according to claim 13, wherein, the processing circuit system is configured to: provide a request for user input regarding the control of the second imaging device, and encode the control signal based on the determination of the user preferences of the user with the largest number.

16. A method for remotely controlling an imaging device, the method comprises: capturing an image of the iris of a user of a display device; detecting the gaze direction of the iris based on the image; and encoding a control signal based on the gaze direction to control the position of an imaging device remote from the display device.

17. The method according to claim 16, wherein, the control signal includes data for controlling at least one of a pan function, a tilt function, and a zoom function of the imaging device.

18. The method according to claim 16, wherein, the method comprises: encoding another control signal to control the position of another imaging device.

19. The method according to claim 16, wherein, the method comprises: detecting the gaze directions of a plurality of users.

20. The method according to claim 19, wherein, the method comprises: determining the priority levels of the plurality of users, and encoding the control signal based on the user group with the highest priority.

Images