Computer platform having a shared user experience interface
An integrated computing device with a rotatable display and presence sensor module addresses the challenge of supporting diverse user needs by enabling seamless transitions between ambient and user-specific displays, enhancing family productivity and collaborative interactions.
Patent Information
- Application Number
- KR1020247031609
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-11-15
- Filing Date
- 2022-11-30
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-11-30
AI Technical Summary
Existing computing devices are not configured to adequately support diverse tasks or effectively interface with different family members, leading to issues such as competition for resources, inferior device interoperability, and lack of collaborative interaction.
An integrated computing device with a rotatable display and presence sensor module that detects user presence, enabling personalized experiences through machine learning models, allowing seamless transitions between ambient and user-specific displays and adaptive user interfaces.
The solution provides a single platform suitable for multiple users, enhancing family productivity by facilitating collaborative interactions and personalized experiences without the need for additional hardware or software, while conserving resources.
Smart Images

Figure R1020247031609_ABST
Abstract
Description
Technology Field
[0001] Cross-reference regarding related applications
[0002] The present application claims priority and benefit to the filing date of U.S. Provisional Application No. 63 / 324,302 filed on March 28, 2022, and to the filing date of U.S. Patent Application No. 17 / 987,412 filed on November 15, 2022, the entire disclosure of which is incorporated herein by reference. Background Technology
[0003] People use client computing devices, such as laptops, tablets, and netbooks, in various environments including homes, schools, offices, coffee shops, and airports. Different types of computing devices may be better suited for specific settings or tailored to specific types of applications, such as gaming, computationally intensive computer-aided design, or business drafting tools. In a home setting, a single shared computing device may be used by different family members for a variety of tasks, extending beyond work or gaming to include streaming movies or other content, and children's school assignments. Unfortunately, in many cases, these devices are not configured to adequately support diverse tasks or to effectively interface with different family members.
[0004] The technology relates to a computer platform that addresses technical problems associated with in-home usage, where a single platform is used to support the needs of different users while running different types of applications across a wide variety of scenarios. An embodiment of the technology provides a technical solution as an all-in-one platform in which the computer is integrated within a monitor or other display device. The display device is fixed to a stand or other base and can rotate relative to the base, for example, between a horizontal (or landscape) display mode and a vertical (or portrait) display mode. That is, the integrated housing is rotatably coupled to the stand, allowing the display device to rotate between landscape and portrait modes. Through other interface embodiments related to presence detection and context awareness, the platform can provide personalized experiences for different individuals as well as common features, depending on the contextual user interface. This provides the technical advantage of a single platform suitable for multiple users in many different situations without the need for costly hardware or software add-ons. For example, computer platforms can enhance family or other shared productivity, such as tasks involving more than one person (e.g., homework assignments, group video chatting, watching videos together) or any task that benefits the whole family even if performed by one person (e.g., family scheduling, group note-taking, finding recipes, or paying bills).
[0005] According to one embodiment, a client computing device comprising an integrated housing is provided, wherein the integrated housing comprises: memory configured to store instructions and data for the operation of the client computing device; a display module comprising a display device configured to display visual content; a user interface module operably connected to the display module, wherein the user interface module is configured to receive user input; and a presence sensor module and a processing module. The presence sensor module is configured to detect the presence of one or more people within a threshold distance from the client computing device. The presence sensor module comprises an image sensor, and the presence sensor module is configured to implement one or more machine learning models for detecting the presence of said one or more people in an image captured by the image sensor. The processing module comprises one or more processors. The processing module is operably coupled to the memory, the display module, the user interface module, and the presence sensor module. One or more processors determine, based on information from the presence sensor module, that at least one person has entered within the threshold distance of the client computing device; evaluate the current display state of the display device; Based on an evaluation and a determination that at least one person has entered within a threshold distance, one or more items of content to be displayed on an ambient visual display are selected, and one or more items of content are associated with a shared account of a client computing device; and the display module is configured to display an ambient visual display on a display device according to one or more items of selected content.
[0006] A shared account may include multiple individual user profiles for different users. One or more items of content to be displayed on the surrounding visual display may include one or more selected from a group consisting of a shared calendar, group notes, and a to-do list. One or more items of content displayed on the surrounding visual display may be modified by a person without logging to the client computing device. In one example, the threshold distance is a first threshold distance from the client computing device. Here, a presence sensor module is further configured to detect that a person has entered within a second threshold distance closer than the first threshold distance from the client computing device, and one or more processors are further configured to cause the display module to change from the surrounding visual display to the user visual display based on the person entering within the second threshold distance. The change from the surrounding visual display to the user visual display may include replacing the surrounding visual display with a log-in screen. The change from the surrounding visual display to the user visual display may include modifying the surrounding visual display to highlight one or more specific content features.
[0007] The presence sensor module may further include other sensors configured to detect non-image information regarding whether a person is within a threshold distance of the client computing device. The other sensors may include one or more microphones configured to detect acoustic information within the threshold distance. The other sensors may include a radio frequency ultra-wideband sensor configured to detect the presence of another client computing device within the threshold distance. The ultra-wideband sensor may be configured to detect the pose of another client computing device relative to the client computing device.
[0008] The client computing device may further include a stand, and an integral housing is detachably coupled to the stand. The integral housing is rotatably coupled to the stand so that the display device can rotate between landscape mode and portrait mode. The client computing device may also include at least one orientation sensor configured to detect a change between landscape mode and portrait mode. In one example, the display device includes a touch screen. Here, the client computing device may further include at least one of a keyboard or a mousepad, and one or more processors are further configured to identify the possibility of user input through (i) the touch screen or (ii) the keyboard or mousepad and to adapt actions based on the identification. One or more processors may be configured to identify the possibility based on a change between landscape mode and portrait mode detected by at least one orientation sensor.
[0009] According to any of the above, if it is determined that a person is not detected by the sensor module within a threshold distance for at least a threshold amount of time, one or more processors may be further configured to cause the display module to display a sleep mode screen on the display device. One or more processors may be configured to select one or more items of content based on whether the person detected within the threshold distance has a user profile on the client computing device.
[0010] According to another aspect, a computer-implemented method comprises: detecting the presence of one or more people within a threshold distance from the client computing device by a presence sensor module of the client computing device; determining, by one or more processors of the presence sensor module, that at least one person has entered within the threshold distance of the client computing device based on the detection; evaluating the current display state of a display device of the client computing device by one or more processors; selecting one or more items of content to display on an ambient visual display by one or more processors based on the evaluation and the determination that at least one person has entered within the threshold distance, wherein one or more items of content are associated with a shared account of the client computing device; and causing a display module of the client computing device to display an ambient visual display on a display device according to one or more items of selected content. A shared account may include a plurality of individual user profiles for different users. One or more items of content may include at least one user input item. At least one user input item may be configured to be reformatted, rearranged, or adjusted in response to user input. One or more items of content to be displayed on the surrounding visual display may include one or more selected from a group consisting of a shared calendar, group notes, and a to-do list. Here, one or more items of content displayed on the surrounding visual display may be modified by a person without being logged to an account on the client computing device.
[0011] In one example, the threshold distance may be a first threshold distance from the client computing device. Here, the method further comprises the step of detecting, by a presence sensor module, that a person has entered within a second threshold distance closer than the first threshold distance from the client computing device; and the step of causing a display module to change from an ambient visual display to a user visual display based on the fact that the person has entered within the second threshold distance by one or more processors. The change from the ambient visual display to the user visual display may include replacing the ambient visual display with a log-in screen. The change from the ambient visual display to the user visual display may include modifying the ambient visual display to highlight one or more specific content features. In this approach, the method may further comprise the step of detecting that a person has entered within a third threshold distance closer than the second threshold distance from the client computing device; and the step of causing a display module to change at least one feature of the user visual display based on the fact that the person has entered within the third threshold distance by one or more processors.
[0012] The present method may further include the step of detecting a change in a display device between a landscape mode and a portrait mode by at least one orientation sensor of a client computing device. If the display device is touch-sensitive, the present method may further include the step of identifying the possibility of user input through (i) a touch-sensitive display device or (ii) a keyboard or mousepad by one or more processors; and the step of adapting an action based on the identification. The step of identifying the possibility may be based on the change between a landscape mode and a portrait mode detected by at least one orientation sensor.
[0013] The present method may further include a step of causing a display module to display a sleep mode screen on a display device by one or more processors when it is determined that no person is detected within a threshold distance. One or more items of content may be selected based on whether the person detected within the threshold distance has a user profile on the client computing device. Alternatively or additionally, the present method may include a step of detecting the presence of one or more people within a threshold distance from the client computing device and, when it is determined that at least one person has entered within the threshold distance of the client computing device, performing contactless user identification. Here, contactless user identification may include face recognition.
[0014] The step of selecting one or more items of content to be displayed on a surrounding visual display may be based on the identification of multiple people who have entered within a threshold distance of the client computing device. In this case, the present method may further include the step of displaying group-specific information on the display when one of the multiple people logs into the client computing device. Alternatively or additionally, the present method may further include the step of generating non-visual information when it is determined that at least one person has entered within a threshold distance of the client computing device.
[0015] According to another aspect, a non-transient computer program product includes instructions that cause any of the above methods to be performed when executed by one or more processors.
[0016] According to another aspect, a computer-implemented method comprises: detecting a presence within a threshold distance of a client computing device based on sensor data from a presence sensor of the client computing device; applying one or more machine learning models to the sensor data to determine whether at least one user has entered within the threshold distance of the client computing device based on the detection; selecting one or more items of content to display on an ambient visual display based on the determination, wherein one or more items of content are associated with a shared account for the client computing device; and displaying an ambient visual display having one or more items of selected content to a user. A shared account may include multiple individual user profiles for different users.
[0017] These embodiments and other embodiments may each optionally include one or more of the following features.
[0018] The present invention provides a computer program product comprising instructions that cause any one of the methods described herein to be performed when executed by one or more processors. A non-transient computer-readable medium comprising instructions that cause any one of the methods described herein to be performed when executed by one or more processors is also provided.
[0019] The present invention provides a method comprising: detecting a presence within a threshold distance of a client computing device based on sensor data from a presence sensor of the client computing device; applying one or more machine learning models to the sensor data to determine, based on the detection, whether at least one user has entered within the threshold distance of the client computing device; selecting one or more items of content to display on an ambient visual display based on the determination, wherein one or more items of content are associated with a family (or shared) account for the client computing device, and the family account includes a plurality of individual user profiles for different users; and displaying an ambient visual display having one or more items of selected content to a user. A client computing device configured to perform the present method and comprising a presence sensor is also provided. Brief explanation of the drawing
[0020] FIGS. 1a to 1c illustrate examples related to presence detection according to the mode of the technology. FIGS. 1d to 1f illustrate examples related to shared content according to the mode of technology. FIGS. 1g to 1h illustrate examples of adaptive UI based on display orientation according to the mode of technology. FIG. 2 illustrates a block diagram of an exemplary integrated client device according to the mode of technology. Figure 3 illustrates a functional diagram of an exemplary client device according to the mode of technology. FIG. 4 illustrates an exemplary scenario image evaluation by an existing sensor module for use with an embodiment of the technology. FIGS. 5a through 5d illustrate exemplary presence detection scenarios according to the mode of the technology. Figure 6 illustrates an example of a family account according to the mode of technology. FIGS. 7a and FIGS. 7b illustrate a system for use with an embodiment of the technology. FIG. 8 illustrates a method according to an embodiment of the technology. Specific details for implementing the invention
[0021] outline
[0022] Every member of a household may have one or more computing devices used for specific tasks, such as streaming movies on a tablet, performing assignments or work on a laptop, or interfacing with social media on a mobile phone or smartwatch. However, individual devices may not interoperate and may compete for the same resources, such as WiFi bandwidth. Furthermore, they are often associated with only one user (e.g., a single user profile), which means that certain collaborative interactions among household members may be difficult or impossible to achieve with these devices. For example, this may involve complexity regarding account sharing, login friction, and data shared between accounts within the same device. In contrast, an integrated all-in-one platform, such as the one discussed herein, can easily resolve technical issues including competition for resources, inferior device interoperability, and the lack of collaborative interaction while still enabling personalized experiences.
[0023] General architectures are discussed, and specific features providing technical solutions for different types of situations are explored. These include a presence-based integrated desktop experience, a family space shared as part of an ambient visual display that can easily transition into a personalized experience for each household member, as well as seamless switching related to user interface elements when the display device is rotatable. Initial examples of these are presented, followed by further exploration of the features.
[0024] According to one aspect of the technology, a presence sensor can detect whether a person is present in or near an integrated computing device. This can be accomplished using a dedicated presence sensor isolated from the rest of the processing system, or one integrated with an embedded webcam or other integrated camera unit. The presence sensor captures non-image information, such as sound, heat, radiation information, and / or other information suitable for determining the presence of a person, as well as images, and can evaluate this information, for example, by one or more machine learning (ML) models, to identify whether one or more people are within the field of view of the presence sensor or within close range to the device. If at least one person is identified, the integrated computing device can surface specific information based on the current context, enabling the person(s) to interact with the device in a selected manner. Thus, the computing device can evaluate the current display state of the display device and, based on the evaluation, select one or more items of content to display to the person(s). One or more items of content may include at least one user input item. For example, at least one user input item may be configured to reformat, rearrange, or adjust in response to user input, for example, to move on the display or to change its size or configuration. The current display state may include the content currently being displayed, other current context signals (date, time, calendar information, etc.), or any other information associated with the device state, such as whether the device is locked or logged in.
[0025] FIG. 1a illustrates an example (100) illustrating an integrated client computing device (102) that can be used as a home computer. In this example, the display (104) displays a screen saver (106) because the client device is not actively used. The display (104) may have an aspect ratio such as 16:9, 4:3, or other ratios. The display may have a diagonal size, for example, 19 inches, 21 inches, 24 inches, or larger or smaller. The display may be configured for haptic input using a stylus or touch input with the user's finger(s). As illustrated, the integrated client device may employ a wireless keyboard (108) and / or one or more trackpads or mousepads (110), which may be part of a single unit, or the keyboard may be separate from the trackpad / mousepad or mouse-based input (110a). The integrated client device has a stand (112). One or more microphones (114) may be positioned along the stand or along the housing of the integrated client device (102). The integrated client device may be fixedly mounted on the stand (112), but in an alternative configuration, the screen (having the integrated component) may be detached from the stand, allowing the user to carry it around the house and use it remotely based on battery power. That is, the client device may include an integral housing coupled to the stand (112) (optionally removable or detachable).
[0026] A webcam or other integrated camera (116), which may include a privacy shutter or other features to disable image capture, is positioned along the housing of the integrated client device (102) and may be used for video conferencing, interactive games, etc. An indicator (118), such as an LED, may be turned on to warn the user whenever the webcam (116) is in use. The integrated client device may include a separate camera or other imaging device (120) that is part of a presence sensor. As illustrated, the webcam (116) and the imaging device (120) may each be positioned along the top bezel of the integrated client device housing. In some examples, these devices may be positioned at different locations along the integrated housing. The integrated camera (116) may be used as part of the presence sensor instead of or in addition to the imaging device (120). That is, the presence sensor includes an image sensor configured to capture one or more images. The presence sensor may be configured to detect the presence of one or more people within a threshold distance from the client computing device. For example, the presence sensor includes an image sensor as discussed herein, and the client device is configured to detect the presence of one or more people in images (images) captured by the image sensor.
[0027] In the example illustrated in FIG. 1b, a person (122) enters a room (or environment) where an integrated client device is located. When the person enters within the detection range of an imaging device or image sensor (which may be a webcam or a dedicated presence sensor), within the field of view of the imaging device, and within a certain (first) threshold distance from the integrated client device (e.g., within 2 to 5 meters from the integrated client device), the presence sensor unit or module evaluates one or more images acquired by the imaging device (such as according to one or more ML models implemented by the processing module of the presence sensor). The threshold distance may be a predetermined threshold distance, may be a fixed threshold, or may be modified / configured by the user based on one or more preferences.
[0028] Then, when it is determined that a person is present, the presence sensor causes the operating system of the integrated client device to wake the device. This may include a change from a screen save in sleep mode to an information screen (124) for waking in access mode. Sleep mode may be any low-power mode of the client device's display, such as when the screen is dark or dimmed, or when most of the display device pixels are not driven (off or dimmed). In this mode, the power consumption of the device is reduced compared to full-on or wake mode. By waking the device in access mode as described herein, the client device's resources may be further conserved by not activating the display when no one is near the device. As illustrated in this example, the information screen (124) may be a login screen, but may display other types of information such as a shared (e.g., family) calendar (126), group memos, or a to-do list as illustrated in FIG. 1c. This type of information is surfaced on an (ambient) information screen (or ambient visual display), which allows for asynchronous family scheduling with increased visibility to any family member. In addition to displaying visual information on the display, the system may also generate non-visual information when it is determined that at least one person has entered within a certain threshold distance of the client computing device. This may include playing a tone, tune, or other sound through the speaker(s) of the client device. Alternatively or additionally, this may include turning on or changing the lighting state of one or more peripheral devices, such as a keyboard or mouse.
[0029] For example, when a person enters within a threshold distance of the client device, the content displayed on the ambient visual display is content associated with the client device's family (or group) account. In one scenario, when the user is nearby but not within arm's reach of the integrated client device (e.g., not within 3 to 4 feet), the display may show a calendar or other interesting ambient screen (ambient visual display) to make the device useful at a mid-range distance. Then, when the user is within a second threshold distance, for example, at a close range (e.g., within arm's reach of 3 to 4 feet), the system may disable the ambient screen / screensaver and instead surface the login screen. Here, using biometric login, this can create a highly streamlined login experience. The login screen can be an example of a user visual display (user-oriented or user-specific compared to general or family-oriented ambient visual displays). Conversely, if a person walks away from the integrated client device, the presence sensor can cause the user interface to save or close the active application, display the lock screen, return to the ambient visual display, put the device into sleep mode, etc. Here, this may include a determination by the presence sensor that no person has been detected within a certain threshold distance (e.g., 3 to 5 meters) for at least a threshold amount of time (e.g., 1 minute or more).
[0030] FIG. 1d illustrates another scenario (140) regarding a shared family space. Here, the user interface displays a login (142), which may include separate accounts for each family member (multiple individual user profiles for different users) as well as a common or shared account for the family (a common family or shared account). Since individual user profiles may be embedded in the family account or otherwise associated, the family account includes multiple individual user profiles. When a person logs in as an individual, that user's user experience may be customized to that individual's preferences, including available apps, how apps are displayed, and preferences associated with apps. The family account may be associated with a set of common apps or other features that may surface in the UI when logged in as a family account. This may include a common calendar (see 126 in FIG. 1c), a shopping list (144) illustrated in FIG. 1e, a set of movie options available for streaming (146) illustrated in FIG. 1f, and / or other shared information. Alternatively, even when logged in as an individual, the user interface may still display some or all of the common family information. In one example, the common account may be used by multiple family members. In another example, the common account may be shared by roommates, suitemates, or officemates, for instance, in a dormitory, school, or office setting.
[0031] As mentioned above, according to another embodiment, the display device may be rotated between a horizontal (or landscape) display mode and a vertical (or portrait) display mode, for example, as illustrated in views (160 and 170) of FIG. 1g and FIG. 1h, respectively. Depending on the display mode, user(s) may interact with the integrated client device in different ways. For example, in landscape mode, the user may work on a spreadsheet or write semester assignments. In this case, interaction with the device may focus on input via a keyboard and / or mouse. However, if the screen is rotated to portrait mode and the user is running a paint or computer-aided design app, interaction may be primarily touch-based. Thus, depending on one embodiment, display orientation, usage patterns, application usability, and / or other factors, the system may predict user actions based on specific input types (or identify the possibility of user input through specific input types) and adapt the user interface accordingly to facilitate more robust engagement with the system.
[0032] Integrated System Architecture
[0033] FIG. 2 illustrates a block diagram of an exemplary integrated client device (200), such as a desktop-type client computer. As illustrated, the client device includes a processing module (202) having one or more computer processors, such as a central processing unit (204) and / or a graphics processor (206), as well as a memory module (208) configured to store instructions (210) and data (212). The processors may or may not operate in parallel and may include an ASIC, a controller, and other types of hardware circuits. The processors are configured to receive information from a user through a user interface module (214) and to display information to the user on a display device of a display module (216) through the user interface module. For example, during different modes at different times or in different situations, the user interface module may cause the display device to display a screen saver, an ambient visual display based on user presence, a login screen, as well as apps and other features. The display module (216) has a display interface and can be configured as a touchscreen that enables user input through a stylus or other tool or by the user physically touching the screen.
[0034] As mentioned above with respect to FIGS. 1a through 1h, the integrated client device (200) may include a base (or other stand) and a display housing. According to an aspect of the art, the display housing may include a display module and, together with some or all of the other components shown in FIG. 2, a processing and memory module. In one example, a power module (226) may include a battery for supplying power to the integrated client device when the display housing, which includes the other components, is physically removed from the base for portable use. When connected to the base or other stand, the display module (216) may be configured to pivot, pan, swivel, and / or rotate relative to the base. For example, the display module (216) may rotate between a horizontal (horizontal) orientation and a vertical (vertical) orientation, as shown in FIG. 1g and FIG. 1h.
[0035] The user interface module (214) is configured to receive user input. The user interface module (214) can receive commands from the user via user input and convert them to submit to a given processor. The user interface module may be linked to a web browser (not shown). User input may include a keyboard, keypad, mousepad and / or touchpad, microphone, gesture-based input, or other types of input devices, in addition to or alternatively including the touchscreen mentioned above. The keyboard, keypad, mousepad and / or touchpad may be part of or connected to the base via a cable or other wired connection, or may be physically separated from the integrated client device and configured to be connected via one or more wireless connections such as Bluetooth™, WiFi, ultra-wideband (UWB), infrared, etc. The user interface module (214) may be operably connected to a display module (216).
[0036] The display module (216) may include circuitry for driving a display device to display graphics and other information to a user. That is, the display device is configured to display visual content. For example, graphic information may be generated by a graphics processor(s) (206), and the CPU (204) manages the overall operation of the integrated client device (200). The graphic information may display a response to a user query on the display module (216). For example, the processing module may execute a browser application or other service using instructions and data stored in the memory module (208) and display information associated with the browser application or other service to the user through the display module (216). The memory module may include a database or other storage for browser information, location information, etc.
[0037] The memory module (208) may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. The memory module (208) may include, for example, flash memory and / or NVRAM and may be implemented as a hard drive or a memory card. Alternatively, the memory module (208) may also include a removable medium (e.g., a DVD, CD-ROM, or USB thumb drive). One or more regions of the memory module (208) may be writable, and other regions may include read-only (or otherwise write-protected) memory. In one embodiment, a computer program product is tangibly implemented in an information carrier. FIG. 2 functionally exemplifies the processor(s), memory module, and other elements of the integrated client device being within the same whole block, but these components may or may not be stored within the same physical housing. For example, some or all of the instructions and data may be stored on an information carrier, which is a removable storage medium connectable to a base or display housing (e.g., an optical drive, a high-density tape drive, or a USB drive), and other instructions and data may be stored within a read-only computer chip integrated into the base or display housing.
[0038] Data (212) may be retrieved, stored, or modified by the processor according to instructions (210). For example, data may be stored in a computing device register, a relational database, an XML document, or a flat file as a table having multiple different fields and records. Data may also be formatted in any computing device-readable format.
[0039] Instructions (210) may be any set of instructions executed directly (as machine code) or indirectly (as scripts) by processor(s). For example, instructions may be stored as computing device code on a computing device-readable medium. In this regard, the terms “instruction” and “program” may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by processor(s), or in any other computing device language including a script or a collection of independent source code modules that are interpreted or pre-compiled as required.
[0040] Also, as illustrated in FIG. 2, the integrated client device (200) includes a communication module (218) for communicating with other devices and systems, including other client devices, servers, and databases. The communication module (218) includes a wireless transceiver, and alternatively, the module may include a wired transceiver. The integrated client device (200) may communicate with other remote devices through the communication module (218) using various configurations and protocols, including near-field communication (NFC), Bluetooth™, Bluetooth™ Low Energy (BLE), UWB, or other ad-hoc networks, the Internet, intranet, virtual private network, wide area network, local network, private network using a communication protocol proprietary to one or more companies, Ethernet, WiFi, HTTP, and combinations of the above.
[0041] Additionally, the illustrated exemplary device includes one or more position and orientation sensors (220). The position and orientation sensors (220) are configured to determine the position and orientation of one or more parts of the integrated computing device (200), such as a display housing, relative to the base. For example, these components may include a GPS receiver for estimating the latitude, longitude, and / or altitude of the integrated client device, as well as other direction / velocity detection devices such as an accelerometer, gyroscope, or inertial measurement unit (IMU) capable of determining the orientation of the display housing relative to the base (as well as the rate of change of the position of the display housing). The integrated client device (200) may also include one or more camera(s) (222) for capturing still images and recording video streams, such as an integrated webcam and / or dedicated imaging device for presence detection as discussed above. The device may also include one or more microphones (223) (e.g., which may also be used for presence detection by detecting acoustic information within a threshold distance from a client device), a speaker (224), as well as a power module (226). An actuator for providing tactile feedback or other information to a user may be integrated into the touchscreen of a display module (not shown).
[0042] The integrated computing device (200) may perform presence detection using an integrated camera, but in some configurations, the device may include a separate presence sensor module (228). The presence sensor module (228) may be configured to detect the presence of one or more people within a threshold distance from the computing device (200). As illustrated, such a module includes an image sensor (230), a processing module (232), and a memory module (234). In one example, the image sensor may be a dedicated low-resolution camera capable of providing grayscale or color (e.g., RGB) images having a size (in pixels) of 320 x 240, 300 x 300, or similar (e.g., + / - 20%). During operation, images may be captured by the image sensor (230) (or webcam) once every 2 to 10 seconds (or more or less). The presence sensor module may be configured to implement one or more machine learning (ML) models to detect the presence of one or more people in an image captured by the image sensor (230). The processing module may include an FPGA or other processing device capable of processing images received from the image sensor in real time using one or more ML models. The model itself may be stored in the memory module (234) or main memory (208) of the device (200).
[0043] The presence sensor module (228) may be configured to operate using as little power as possible, for example, approximately 100 mW or less. In one scenario, the module (228) may use 5 to 10 mW in "Wake on Approach" mode (as in the example of FIG. 1a and FIG. 1b). This can be achieved by turning off the processing module (232) while it is not needed. Here, for example, the module (228) may wake up the processing module (232) by relying on motion detection integrated into the image sensor, or start the processing module (232) once every approximately 1 to 15 seconds to check if it is needed. Images acquired by the image sensor for presence detection may not be stored in memory after processing. Regardless of whether any images are retained by the presence sensor module, they may not be transmitted to other parts of the device to be used as images for the webcam.
[0044] FIG. 3 illustrates an example (300) of a logical architecture of an integrated client device (200) having a presence sensor module (228).
[0045] As illustrated, the controller (302) of the presence sensor module (228) may be operably connected to the image sensor (230), the processing module (232), and the memory module (234). One or more other types of sensors (236) (e.g., a microphone for audio or acoustic detection or a radio frequency UWB sensor configured to detect nearby client devices) may also be provided. The controller (302) may be configured to turn the processing module (232) and the memory module (234) on and off, and may update the memory as needed, such as adding a new ML model or updating an existing model. The controller (302) may be coupled to the image sensor via an I2C interface (303) and may be coupled to the processing and / or memory module via an SPI interface. The controller (302) may be responsible for managing the power state and transmitting configuration and status from and to the integrated client device operating system.
[0046] The image sensor of the presence sensor module (or the computer's integrated camera) is configured to output an image to the processing module (232) and can transmit motion detection information to the controller (302), but cannot transmit an image. As mentioned above, another (non-image) sensor (236) may be integrated into the presence sensor module (228). For example, this may include a microphone (or microphone array) (236a) and / or a UWB sensor (236b). The microphone(s) can pick up sound, while the UWB sensor can detect the presence of other nearby UWB-enabled computing devices. In the latter case, for example, the UWB sensor may be used to identify the pose of another device, such as how far another UWB-enabled device (e.g., a mobile phone or tablet computer) is from the integrated client device, as well as the relative angle at which the other device is oriented and how much the device is currently tilted (e.g., whether the other device is pointing toward the integrated client device, such as a remote control). Information from the sensor (236a and / or 236b) may be combined with image-based information to, for example, verify image information or confirm the presence of a specific individual.
[0047] For example, the presence sensor module may be defaulted to a very low power state that is only looking for motion (and / or the presence of sound or other UWB-enabled devices). The presence sensor module may be configured to initially detect the presence of one or more people within a threshold distance from the device. When presence / motion is detected by the image sensor (or when other presence information is received), other components of the presence sensor module (228) may be powered on to determine if there are people nearby. For example, after initial motion (or presence) detection, the processing module (232) of the presence sensor may be configured to apply or implement one or more machine learning models to detect the presence of one or more people in an image captured by the image sensor. That is, the processing module (232) will start a presence detection run to determine whether the device needs to be fully woken up.
[0048] If it is determined that there is no person nearby (i.e., the initial presence detection was erroneous, or was not caused by a person), the system may return to low-power motion detection. Otherwise, the processing model of the client computing device may determine that at least one person has entered (is) within the threshold distance of the client computing device based on information from the presence sensor module. This two-step approach to detecting at least one person within the threshold distance of the client computing device may help save computational resources. The processing module (232) may temporarily store data in local memory when running one or more ML models on the received image. Once processed, the processing module (232) is configured to transmit commands and / or data to the controller (302). For example, commands transmitted to the microcontroller may include (1) person detection; (2) no person detection; or a second (or additional) person detection, etc. These commands may be transmitted to the operating system of the integrated client device with minimal additional processing.
[0049] As illustrated, the controller (302) is operably coupled to the operating system (304) of the integrated client device. For example, this can be done using an I2C or SPI bus interface. Through these interfaces, the controller (302) can issue an interrupt (306) or transmit a command, result, or other signal (308) via the bus (310), which can be used by the operating system, a specific app or program (e.g., a login app, a family welcome screen, or a video conferencing program) or another part of the integrated client device to take some action when it is determined that one or more people are in the field of view of the imaging device of the presence sensor (or detected by other sensors of the presence sensor). For example, the interrupt may indicate the presence of a person or other conditions of the environment that may be detected by the presence sensor module. The interrupt may be used to wake the computing device from pause or standby mode, for example, to initiate face authentication, or to display information such as family notifications, a shared calendar, or local weather. Therefore, a general mode of operation allows the presence sensor module to transmit inference results for one or more models executed by local processing to the operating system, and enables one or more processes of the operating system to interpret the results and respond or otherwise proceed accordingly.
[0050] As illustrated, the operating system may logically include one or more routines or other processes (318), such as a kernel (312), a presence sensor daemon (314), firmware (316), and controlling power to a display device or other device. In this example, the presence sensor daemon (314) is a software daemon responsible for coordinating communication between the presence sensor module (228) and the process (318). The kernel may communicate with the routine or other process via a system bus (320), such as a d-bus, for inter-process communication.
[0051] Presence detection
[0052] According to the mode of the technology, for presence detection, the system may employ one or more ML models stored in memory. For example, the models may include one model for detecting whether a random person is present and another model for detecting whether any other random person is nearby. Identification of multiple people may be triggered by a specific program running on an operating system or an integrated client device to display specific family-related information (e.g., shared calendar, shopping list, chore list, etc.) in a user interface or to include specific family-related information when an individual logs into the device.
[0053] The model implemented in the presence sensor module is configured to detect human faces and / or other parts of a person in an image. For example, while the head may be mostly over the screen, the person's torso, arms, or other parts of the body may be visible. Thus, although a cat or other pet may approach the client device, the model is designed so that the system does not react to the presence of such non-humans. In some examples, if at least one person is detected to have entered within a threshold distance of the client computing device (detected to be inside), the content displayed on the surrounding display may be content associated with a family account for the client computing device. However, in some implementations, human identification may involve determining whether one or more faces are "registered" with the integrated client device. In one scenario, in addition to face detection, the system may be configured to perform face recognition. Here, one or more models may be trained based on specific family members. Therefore, in the example, the system may display specific family-related information (e.g., a calendar indicating upcoming doctor appointments or dental visits) on the surrounding visual display when it is confirmed that person(s) are registered, but if one or more people are not registered, the surrounding visual display may not surface family-related information (e.g., displaying a general weekly calendar or clock instead). In other words, content associated with the family account on the client computing device is displayed on the surrounding visual display only when the detected person is registered as part of the household / family.
[0054] In one scenario, the model(s) can reliably detect a face approximately 2 to 3 meters away from the camera with a face width of about 10 pixels. In this scenario, the model(s) can also satisfy the following requirements. First, they must be able to work with grayscale (or color) images having an aspect ratio such as 320x240 or 300x300. Each model may employ, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) network, or a combination thereof.
[0055] In one scenario, model training may include one or more types of image augmentation to help create a robust model. This may include face size scaling (e.g., helping to distinguish between children and adults or to determine whether someone is close or far away), face translating, face clipping, multi-face image synthesis, and / or blurring selected face details. In the baseline set of images, random images without faces may be discarded before training. Then, depending on one example, one or both of the following are performed: (i) select the largest face from the image set and scale it so that the face height is 110% of the image height, and (ii) move the face to a random location in the image. In this case, at least 60 to 80% of the image must be visible on the screen (not clipped at the edges). If padding is required in the image due to translating, the last row / column may be repeated or the image may be filled by reflection (care not to remove / reflect other faces). Training may also include adding a second person for "synthesis". Here, two images containing faces are selected. Then, one of the faces is smoothly blended into the other image. This can integrate areas that also include parts of the body. Since the blend must look as realistic as possible, the ML model cannot learn that the image containing the blended face is always a second person. This can also be aided by blending the face into an image that does not contain any faces.
[0056] FIG. 4 illustrates an exemplary scenario (400) for image evaluation by an existing sensor module in the above view. In block 402, the image sensor captures an image that may be a grayscale or color image. The image may be, for example, 320 x 240 or 300 x 300 or similar in size, and may have a medium or low resolution. Next, in block 404, one or more pre-processing operations may be performed by the image sensor or the processing module. For example, the image sensor or the processing module may adjust image exposure, gain (analog and / or digital gain) or other image parameters, crop the image, and convert from a color (e.g., RGB) image to a grayscale image if the image sensor (e.g., an image sensor with a Bayer filter) generates a color image. Pre-processing may additionally or alternatively include detecting cases where the image is unusable (e.g., all black or fully saturated).
[0057] Image pre-processing may additionally or alternatively include the image sensor extracting motion information from the image based on, for example, a comparison of the image pixels with the pixels of one or more images taken immediately prior to the current image. Here, the extracted motion information may be transmitted from the image pre-processing block (404) to the controller as indicated by the dotted downward arrow. Note that image sensor parameters, such as taking into account different lighting conditions, may be calibrated in the initial settings (e.g., whenever the human presence sensor module is turned on or whenever the image sensor is initialized) and may also be adjusted between image captures. For example, during the first image capture, there may be no one in the room and the lights may be off. However, if a person enters the room and turns on the lights, this may require adjustments to the exposure or other parameters. The image capturing process itself may occur continuously every X milliseconds, such as every 100 to 500 milliseconds (or more or less), as long as the presence sensor module is operating. Here, the operation of the module may include the user of the integrated client device positively granting authorization.
[0058] In block 406, local processing applies one or more ML models to a raw or pre-processed image. The models are maintained in local memory, and during processing, data is temporarily stored, for example, in the RAM of a memory module. In addition to detecting the presence (or absence) of one or more people in the image, there may be one or more models configured to detect other parts of a person, such as a human face or torso, arms, hands, legs, etc. Other models may be trained to detect a person wearing a mask or a person whose face is partially obscured in a different way (such as when the person is not looking directly at the image sensor). As illustrated by block 408, the output from the applied model may be an indication that one or more people are present. And as illustrated by arrow (410), an interrupt, command, result, or other signal may be issued by the local processing and / or module controller so that the operating system or other part of the client device can perform one or more actions in response to the presence detection.
[0059] Exemplary Scenarios and Applications
[0060] In recent years, in-home computing has become much more relevant and widespread in both workplaces and schools. The experience of how people access computers, how they reach their desired destinations simply and quickly, and how devices feel personalized and relevant to each individual is becoming increasingly important. These issues were previously not adequately addressed for home desktop-type systems in many cases, particularly in households where each individual has specific computing needs that often differ from those of others in the family.
[0061] Technical solutions for this include modalities related to shareability, context, and continuity. Shareability provides a multi-user experience suitable for an entire family or different households. For example, an integrated client device can be placed in a living room or other common space accessible to all family members. Context allows computing systems to provide situation or scenario-specific information to user(s), thereby customizing user interface interactions and enabling useful transitions. Continuity enables the integrated client computer device to seamlessly transition content interactions with other user devices, such as a television in the same room for streaming movies, a mobile phone or tablet for sharing video clips, or an appliance displaying recipes and controlling other devices like a thermostat. Thus, content can be cast to or from other devices for an optimized user experience, such as casting a movie from an integrated client device to a high-definition large-screen television, posting video clips from a mobile phone to the integrated client device's display, or sharing photos to the display of an appliance in another room. Another continuity-related solution is session synchronization between computers. If there is another computer in addition to the unified client computing device, the unified client computing device can transfer open windows, apps, and / or workflows from one device to another. For a family member who is a student working on an assignment on a laptop and wishes to use a larger screen, accessing the unified client computing device will mirror the assignment window displayed on the laptop's display or migrate it to the unified client computing device's display screen.More generally, when an all-in-one computer functions as an auxiliary device, continuity can be important, so transitions to and from other computing devices can be very beneficial.
[0062] Existence-based adaptive experience
[0063] According to one aspect of the technology, presence detection enables an integrated client device to adapt to people and situations. For example, as illustrated in view (500) of FIG. 5a, the display may be turned off or a screen saver (502) displayed when no one is nearby. In accordance with applicable privacy guidelines, if presence detection is used to determine that someone has entered the room or is within a first threshold distance from the computer, the display changes. As illustrated in view (510) of FIG. 5b, this may involve turning on the display when one or more people enter the room and displaying relevant (ambient) information (512) (an ambient visual display showing one or more content items), such as a shared calendar and / or grocery list. Thus, a person can simply walk to the computer, wake it up, and see what is on the ambient screen. In this way, the content is there when the person looks. The information on the ambient screen may be visible to everyone whose presence has been detected (e.g., without employing facial recognition to identify if a specific person is in the room).
[0064] In one scenario, the interface displays a selection of useful visual content based on the current context. For example, a calendar can display upcoming items such as the next day, the next few hours, or the weekend. Other context-specific information, such as a grocery shopping list or a school supply list, may surface at a specific time or serve as a reminder that the activity will occur at a specific time or date. Thus, a shopping or school supply list might be displayed on Saturday morning when the activity is planned for that date, but not on Friday night. Instead, on Friday night, the ambient visual display might display a notification for a family movie night, including details about a specific movie or a list of new releases available for streaming. In other words, the client device evaluates the current display state of the display device and, based on that evaluation, selects one or more content items to display on the ambient visual display. In these examples, one or more content items are associated with a family account for the client computing device. In some examples, one or more content items displayed on the ambient visual display can be modified by a person (or user) without being logged to the client computing device. The content can be modified by voice control or any other user input.
[0065] Then, in some examples, when a person enters within a second, closer threshold distance detected by a presence sensor, the ambient display may be adjusted or changed to a more active content mode (a more specific user visual display). For example, as illustrated in view (520) of FIG. 5c, the display (522) may be switched to a login screen that allows the user to log in to their own account or a shared (family) account. That is, the change from the ambient visual display to the user visual display may involve replacing the ambient visual display with a login screen. Here, if facial recognition or other contactless user identification is performed, the user may be automatically logged in without going through the process on the login screen. In other examples, the change from the ambient visual display to the user visual display may involve modifying the ambient visual display to highlight one or more specific content features. For example, this may include visually highlighting selected content from surrounding visual displays, such as specific appointments for the person (e.g., a doctor's appointment or a dental visit), reminders for scheduled meetings or homework deadlines, or items on a family to-do list intended for the person.
[0066] The reverse interface approach may occur when the user moves away from or leaves the room, such as by first switching to a surrounding display and then switching to a screen saver or another lock screen, or by switching directly to the lock screen. For example, the first threshold distance may be between 2 and 5 meters from the computer or when entering the room, and the second threshold distance may be within 1 to 2 meters from the computer. Other closer and / or closer or farther distances may be employed. Additionally, more than two threshold distances may be evaluated. For example, information displayed may change when moving from a second threshold distance of about 1 to 2 meters from the computer to a third threshold distance in front of the computer. For example, this may include changing at least one feature of the user's visual display from a login input screen to a face recognition screen (if it is a feature enabled for face recognition). Additionally, in one example, the path taken by the person may be a factor when displaying the surrounding screen relative to the active mode where the login or other content is displayed. Alternatively or additionally, when UWB sensors are employed, the way other computing devices are maintained or oriented relative to the integrated client device may also affect the display of the surrounding screen relative to the active mode in the active mode. In one example, based on details and / or other information observed by a presence sensor, the system may select specific content to display when a person enters within a second threshold distance.
[0067] In one embodiment, subject to applicable privacy guidelines, the system may differentiate the content to be displayed based on the identification of a specific user. For example, this may include pre-selecting a person's account from among several different individual profiles included in the family account on the integrated client device when that person approaches. For example, this may include visualizing selected content or surfacing more calendar event details for that registered user, such as providing specific details of an appointment for that person (e.g., a doctor's appointment or a dental visit), reminders for scheduled meetings or homework due dates, or items on a family to-do list intended for that person.
[0068] In other scenarios, specific surrounding information may continue to be displayed even when the person is within a second threshold distance. For example, a shopping list, calendar, or set of notes may be accessible for anyone to view and edit without logging into the computer. Here, modifications to such items may result in updates being transmitted to other user devices (e.g., via cloud sharing approaches or local wireless transmission via Bluetooth™, WiFi, or cellular links). Thus, family or shared calendars, shopping lists, notes, etc., can be set as interactive screensavers in the surrounding settings. This allows the entire family to quickly be reminded of upcoming milestones on a shared device. Content or information may be modified without logging into the device. Here, according to one embodiment, since the display may show family-related information while logged out or locked, positive permission (such as voice, facial, or other biometric permission) may be required to display this information.
[0069] In additional scenarios, selected ambient information may include a night screen that surfaces only the most relevant information (time, sunrise time, alarm time, etc.). This can display information with a low-light approach where only specific information is illuminated against a dark background, as illustrated in view (530) of FIG. 5d. In another scenario, the user may use auxiliary features while the display is in ambient or lock screen mode, particularly for tasks that do not rely on the user profile. This may include, for example, setting a timer (e.g., a movie for a family fun night), unit conversion, or asking long questions to other family members. The user may use voice to ask these questions and receive answers. Identification by the system of the user's presence may enable auxiliary features to provide targeted assistance. And in yet another scenario, the integrated client device may be employed as a "watchtower" or alarm at night. For example, after a certain amount of time has passed, the detection of a human presence (unidentifiable) may cause a push notification to a family phone or other device, sound, or otherwise convey to one or more family members that a human presence has been detected.
[0070] Shared family space
[0071] While different family members may possess individual devices such as laptops, tablets or netbooks, or mobile phones, for an integrated client device that functions as a desktop at home for family sharing, it may be fundamentally important to create a virtual space that is easily shared among family members while allowing for an intuitive and easy transition to each family member's personalized experience. Thus, one modality involves family-level context and information sharing, whereas the other modality involves easy and seamless context switching between shared and individual experiences.
[0072] For example, as illustrated in view (520) of FIG. 5c, each individual user may have their own profile, and each profile may have its own set of preferences. This may include user-specific information regarding email, content streaming, social media, games, etc. However, according to one embodiment, a family account may be created, and a set of family member profiles for each user of the computer may be included in the family account. In this way, the user interface of the computer may provide relevant shared information that is intentionally shared among family members. FIG. 6 illustrates an example (600) of a settings screen for a family account.
[0073] Here, the system can align with a multi-profile experience where each profile is owned by a family member and the main account is for the entire family. This provides family members with quick and easy access to personalized experiences without switching accounts. Shared information associated with the shared family account can be intuitively displayed through features such as a surrounding screen calendar, a family profile web browser, and / or applications accessed via the family profile (e.g., streaming services or other entertainment accounts, digital family photo albums, etc.).
[0074] Seamless switching for human-computer interfaces
[0075] As mentioned above, the display for the integrated client device can rotate, for example, from a landscape orientation to a portrait orientation. That is, the integrated housing can be rotatably coupled to a stand / base. Historically, computing devices with touchscreens (e.g., tablets or mobile phones) have tended to have fairly consistent interaction modes, which makes it quite simple to predict how a user will want to use the device. However, the technical issue with desktop-type computers with touchscreens is that there are at least two related modalities: touch and keyboard / mouse, while voice commands and / or gestures potentially provide additional modalities. The usefulness of these modalities can vary depending on the user, use case, and context. For this reason, having a seamless transition between the two experiences becomes critical to ensuring that the user experience feels natural and fluid.
[0076] According to one embodiment, the system can identify, or otherwise anticipate, the likelihood of user behavior being mouse / keyboard versus touch based on user patterns, app usability, or general user journey insights, and can adapt the UI to facilitate that experience. Processing of one or more systems may also be adapted. That is, a client computing device may be configured to identify the likelihood of user input by a specific type of user input and adapt actions based on that identification. For example, the system may employ one or more machine learning user models to understand when a user is more likely to interact with the screen using touch input than using a mouse or keyboard. This can also be known, for example, by the rotational orientation at the moment based on historical data regarding how the user interfaces with the device. Consequently, the system can prioritize the detection of input through the anticipated input source, thereby providing faster response times for user input. For example, a touch screen typically has a response time of tens of milliseconds to about 100 milliseconds. However, by anticipating touch-based input instead of using a keyboard (or mouse), the system can apply more processing resources to reduce touch screen response time to a maximum of 10 milliseconds (an example of adapting actions). Thus, latency can be reduced, thereby improving the user input mechanism.
[0077] For example, in the case of a large horizontally oriented screen (e.g., a screen with a diagonal size of 24 inches or more), the user interface enables various split-screen modes to separate different apps or programs, such as end-of-semester assignments on one side of the window and a web browser used to search for source materials on the opposite side. Alternatively, a drawing program may have a drawing window on one side and a selectable set of tools in a separate window. Rotating the display to portrait mode may cause the user interface to rearrange where the different windows are displayed (e.g., drawing tools may move from the right window to the bottom window, and the drawing window itself may move from the left window to the top window). In some examples, one or more processors are configured to identify the possibility of a specific type of input based on the change between landscape and portrait modes detected by an orientation sensor. In other examples, an independent specific type of possibility is identified independently of the ability of the integral housing to rotate relative to the base / stand of the client device.
[0078] Exemplary network
[0079] As mentioned above, in some situations, information from an integrated client device may be transmitted to other devices, such as providing calendar updates or conveying other details of a shared family account. An exemplary computing architecture supporting this is illustrated in FIGS. 7a and 7b. In particular, FIGS. 7a and 7b are schematic and functional diagrams of an exemplary system (700) comprising a plurality of computing devices and a database connected via a network, respectively. For example, the computing device(s) (702) may be a cloud-based server system that provides, or otherwise supports, one or more apps, games, or other programs used by the integrated client computing device. The database (704) may store app / game data, user profile information, or other information. The server system may access the database via a network (706). The client device may include one or more of a desktop-type integrated client computer (708), a laptop or tablet PC (710), and an in-home device such as a smart display (712a) and / or a smart thermostat (712b). Other client devices may include personal communication devices such as mobile phones or PDAs (714), or wearable devices (716) such as smartwatches or head-mounted displays (e.g., virtual reality headsets). Other exemplary client devices are large screen displays such as high-definition wall-mounted televisions (718) that can be used in a living room or den during family gatherings.
[0080] In one example, the computing device (702) may include one or more server computing devices having a load-balanced server farm or cloud computing system, which exchange information with different nodes of a network to receive, process, and transmit data to and from other computing devices. For example, the computing device (702) may include one or more server computing devices capable of communicating with any of the computing devices (708 to 718) via the network (706). This may be done as part of hosting one or more collaboration apps (e.g., video conferencing programs, interactive spreadsheet apps, or multiplayer games) or services (e.g., movie streaming services or interactive game shows where viewers can provide comments or other feedback).
[0081] As illustrated in FIG. 7b, each computing device (702 and 708 through 718) may include one or more processors, memory, data, and instructions. Memory stores information that can be accessed by one or more processors, including instructions and data that can be executed by or otherwise used by the processor(s). Memory may be any type capable of storing information that can be accessed by the processor(s), including computing device-readable media. Memory is a non-transient medium such as a hard drive, memory card, optical disc, solid-state, etc. The system may include different combinations of the above, thereby storing different parts of instructions and data on different types of media. Instructions may be any set of instructions executed by the processor(s) directly (as machine code) or indirectly (as scripts). For example, instructions may be stored as computing device code on computing device-readable media. In this regard, the terms "instruction," "module," and "program" may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language including a collection of scripts or independent source code modules that are interpreted or pre-compiled as required.
[0082] The processor may be any ordinary processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC, a graphics processing unit (GPU), a tensor processing unit (TPU), or other hardware-based processor. Although FIG. 7b functionally exemplifies the processor, memory, and other elements of a given computing device being within the same block, such a device may actually comprise multiple processors, computing devices, or memory that may or may not be stored within the same physical housing. Likewise, memory may be a hard drive or other storage medium located in a housing different from the housing of the processor(s), for example, in a cloud computing system of the server (702). Thus, a reference to a processor or computing device will be understood to include a reference to a collection of processors, computing devices, or memory that may or may not operate in parallel.
[0083] The computing device may include all components commonly used in relation to the computing device, such as the processor and memory described above, as well as a user interface subsystem for receiving input from a user and displaying information (e.g., text, images and / or other graphic elements) to the user. The user interface subsystem may include one or more user inputs (e.g., a camera, mouse, keyboard, touch screen and / or microphone facing at least one front (user)) and one or more display devices capable of operating to display information (e.g., text, images and / or other graphic elements). Other output devices, such as speaker(s), may also provide information to the user. And as described in detail above in relation to FIGS. 2 and 3, the integrated client device (708) may include a presence sensor module in addition to the elements described above.
[0084] User-related computing devices (e.g., 708 to 718) may communicate with a back-end computing system (e.g., server (702)) through one or more networks, such as network (706). User-related computing devices may also communicate with each other without communicating with the back-end computing system. Network (706) and mediation nodes may include various configurations such as local in-home networks and protocols including short-range communication protocols such as Bluetooth™, Bluetooth LE™, the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi, and HTTP, and various combinations of those described above. Such communication may be facilitated by any device capable of transmitting data to and from other computing devices, such as modems and wireless interfaces.
[0085] In another aspect of the technology, an integrated client computing device can function as a display for other computing devices. Thus, one or more video-type inputs for a game console or other device (e.g., via cable inputs such as HDMI, VGA, etc., or wireless connections) can be connected to the integrated client computing device. Here, user(s) can access the inputs for the game console or other device through the display instead of using the operating system of the integrated client computing device. Therefore, by supporting other devices through the display, the integrated client computing device can be additionally used to consolidate redundant hardware and provide a clean work or entertainment environment.
[0086] Exemplary method of action
[0087] FIG. 8 illustrates a method (800), the method comprising, in block (802), a step of detecting the presence of one or more people within a threshold distance from the client computing device by a presence sensor module of the client computing device. In block (804), the method comprises, based on the detection, a step of determining by one or more processors of the presence sensor module that at least one person has entered within the threshold distance of the client computing device. In block (806), the method comprises, by one or more processors, a step of evaluating the current display state of the display device of the client computing device. In block (808), the method comprises, based on the evaluation and the determination that at least one person has entered within the threshold distance, a step of one or more items of content to be displayed on the surrounding visual display by one or more processors. One or more items of content are associated with a shared account for the client computing device. And in block (810), the method comprises, a step of causing the display module of the client computing device to display the surrounding visual display on the display device according to one or more items of selected content.
[0088] Although the technology of the present invention has been described with reference to specific embodiments, it should be understood that such embodiments are merely illustrative of the principles and applications of the technology. Accordingly, it should be understood that many modifications may be made to the exemplary embodiments and other arrangements may be devised without departing from the spirit and scope of the technology as defined by the appended claims.
Claims
Claim 1 A client computing device comprises an integrated housing, wherein the integrated housing includes: a memory configured to store instructions and data for the operation of the client computing device; a display module comprising a display device configured to display visual content; a user interface module operably connected to the display module, wherein the user interface module is configured to receive user input; a presence sensor module configured to detect the presence of one or more people within a threshold distance from the client computing device, wherein the presence sensor module comprises an image sensor, and the presence sensor module is configured to implement one or more machine learning models for detecting the presence of the one or more people in an image captured by the image sensor; and a processing module comprising one or more processors, wherein the processing module is operably coupled to the memory, the display module, the user interface module, and the presence sensor module, and the one or more processors determine, based on information from the presence sensor module, that at least one person has entered within the threshold distance of the client computing device; and evaluate the current display state of the display device. A client computing device configured to select one or more items of content to display on a surrounding visual display based on the above evaluation and the determination that at least one person has entered within the above threshold distance, wherein one or more items of content are associated with a shared account of the client computing device; and wherein the display module is configured to display the surrounding visual display on the display device according to one or more items of the selected content. Claim 2 A client computing device according to claim 1, wherein the shared account comprises a plurality of individual user profiles for different users. Claim 3 A client computing device according to claim 1 or 2, wherein one or more items of the content to be displayed on the surrounding visual display include one or more selected from a group consisting of a shared calendar, group memos, and a to-do list. Claim 4 A client computing device according to paragraph 3, wherein one or more items of the content displayed on the surrounding visual display can be modified by a person without logging to the client computing device. Claim 5 A client computing device according to claim 1 or 2, wherein the threshold distance is a first threshold distance from the client computing device; the presence sensor module is further configured to detect that a person has entered within a second threshold distance closer than the first threshold distance from the client computing device; and the one or more processors are further configured to cause the display module to change from the surrounding visual display to the user visual display based on the fact that the person has entered within the second threshold distance. Claim 6 A client computing device according to claim 5, wherein the change from the surrounding visual display to the user visual display includes replacing the surrounding visual display with a log-in screen. Claim 7 A client computing device according to claim 5, wherein the change from the surrounding visual display to the user visual display includes a modification of the surrounding visual display to highlight one or more specific content features. Claim 8 A client computing device according to claim 1 or 2, wherein the presence sensor module further comprises another sensor configured to detect non-image information regarding whether a person is within the threshold distance of the client computing device. Claim 9 In claim 8, the client computing device comprising one or more microphones configured to detect acoustic information within the threshold distance, wherein the other sensor comprises the other sensor. Claim 10 In claim 8, the client computing device comprises a radio frequency ultra-wideband sensor configured to detect the presence of another client computing device within the threshold distance. Claim 11 In claim 10, the client computing device, wherein the ultra-wideband sensor is configured to detect the pose of the other client computing device relative to the client computing device. Claim 12 A client computing device according to claim 1 or 2, further comprising a stand, wherein the integral housing is detachably coupled to the stand. Claim 13 A client computing device according to claim 1 or 2, further comprising a stand, wherein the integral housing is rotatably coupled to the stand so that the display device can rotate between a horizontal mode and a vertical mode; and wherein the client computing device comprises at least one orientation sensor configured to detect a change between the horizontal mode and the vertical mode. Claim 14 In paragraph 13, the display device comprises a touch screen; the client computing device further comprises at least one of a keyboard or a mousepad; and the one or more processors are further configured to identify the possibility of user input through (i) the touch screen or (ii) the keyboard or mousepad and to adapt an action based on said identification, the client computing device. Claim 15 A client computing device according to claim 14, wherein the one or more processors are configured to identify the possibility based on the change between the horizontal mode and the vertical mode detected by the at least one orientation sensor. Claim 16 A client computing device according to claim 1 or 2, wherein if it is determined that a person is not detected by the presence sensor module within the threshold distance for at least a threshold time amount, the one or more processors are further configured to cause the display module to display a sleep mode screen on the display device. Claim 17 A client computing device according to claim 1 or 2, wherein the one or more processors are configured to select one or more items of the content based on whether a person detected within the threshold distance has a user profile on the client computing device. Claim 18 A computer-implemented method comprising: detecting presence within a threshold distance of a client computing device based on sensor data from a presence sensor of the client computing device; applying one or more machine learning models to the sensor data to determine whether at least one user has entered within the threshold distance of the client computing device based on the detection; selecting one or more items of content to display on an ambient visual display based on the determination, wherein one or more items of content are associated with a shared account for the client computing device; and displaying the ambient visual display having one or more items of selected content to the user. Claim 19 In paragraph 18, the computer-implemented method wherein the shared account comprises a plurality of individual user profiles for different users. Claim 20 A computer-implemented method according to claim 18 or 19, wherein the threshold distance is a first threshold distance from the client computing device, and the method further comprises the steps of: detecting by the presence sensor that a person has entered within a second threshold distance that is closer than the first threshold distance from the client computing device; and changing from the surrounding visual display to the user visual display based on the fact that the person has entered within the second threshold distance. Claim 21 A computer-implemented method comprising: detecting the presence of one or more people within a threshold distance from the client computing device by means of a presence sensor module of the client computing device; determining, by means of one or more processors of the presence sensor module, that at least one person has entered within the threshold distance of the client computing device based on the detection; evaluating the current display state of a display device of the client computing device by means of the one or more processors; selecting one or more items of content to be displayed on an ambient visual display by means of the one or more processors based on the evaluation and the determination that the at least one person has entered within the threshold distance, wherein one or more items of content are associated with a shared account of the client computing device; and causing a display module of the client computing device to display the ambient visual display on the display device according to one or more items of selected content. Claim 22 In paragraph 21, the computer-implemented method wherein the shared account comprises a plurality of individual user profiles for different users. Claim 23 A computer-implemented method according to claim 21 or 22, wherein one or more items of the content include at least one user input item. Claim 24 A computer-implemented method according to claim 23, wherein at least one user input item is configured to be reformatted, rearranged, or adjusted in response to user input. Claim 25 A computer-implemented method according to claim 21 or 22, wherein one or more items of the content to be displayed on the surrounding visual display include one or more selected from a group consisting of a shared calendar, group memos, and a to-do list. Claim 26 A computer-implemented method according to claim 25, wherein one or more items of the content displayed on the surrounding visual display can be modified by a person without logging them to an account on the client computing device. Claim 27 A computer-implemented method according to claim 21 or 22, wherein the threshold distance is a first threshold distance from the client computing device, and the method further comprises: a step of detecting by the presence sensor module that a person has entered within a second threshold distance that is closer than the first threshold distance from the client computing device; and a step of causing the display module to change from the surrounding visual display to the user visual display by the one or more processors based on the fact that the person has entered within the second threshold distance. Claim 28 A computer-implemented method according to claim 27, wherein the change from the surrounding visual display to the user visual display comprises replacing the surrounding visual display with a login screen. Claim 29 A computer-implemented method according to claim 27, wherein the change from the surrounding visual display to the user visual display comprises modifying the surrounding visual display to highlight one or more specific content features. Claim 30 A computer-implemented method according to claim 27, further comprising: a step of detecting that the person has entered within a third threshold distance closer than the second threshold distance from the client computing device; and a step of causing the display module to change at least one feature of the user visual display based on the fact that the person has entered within the third threshold distance by the one or more processors. Claim 31 A computer-implemented method according to claim 21 or 22, further comprising the step of detecting a change in the display device between a horizontal mode and a vertical mode by at least one orientation sensor of the client computing device. Claim 32 A computer-implemented method according to claim 31, wherein the display device is touch-sensitive, and the method further comprises the steps of: (i) identifying the possibility of user input through the touch-sensitive display device or (ii) a keyboard or mousepad by the one or more processors; and adapting an action based on the identification. Claim 33 In paragraph 32, the step of identifying the possibility is a computer-implemented method based on the change between the horizontal mode and the vertical mode detected by the at least one orientation sensor. Claim 34 A computer-implemented method according to claim 21 or 22, further comprising the step of, when it is determined that no person is detected within the threshold distance, causing the display module to display a sleep mode screen on the display device by the one or more processors. Claim 35 A computer-implemented method according to claim 21 or 22, wherein one or more items of the content may be selected based on whether a person detected within the threshold distance has a user profile on the client computing device. Claim 36 A computer-implemented method according to claim 21 or 22, further comprising the step of detecting the presence of one or more people within the threshold distance from the client computing device and determining that at least one person has entered within the threshold distance of the client computing device, thereby performing contactless user identification. Claim 37 In paragraph 36, the above-mentioned non-contact user identification is a computer-implemented method comprising face recognition. Claim 38 A computer-implemented method according to claim 21 or 22, wherein the step of selecting one or more items of the content to be displayed on the surrounding visual display is based on the identification of a plurality of people who have entered within the threshold distance of the client computing device. Claim 39 A computer-implemented method according to claim 38, further comprising the step of displaying group-specific information on the display when one of the plurality of people logs to the client computing device. Claim 40 A computer-implemented method according to claim 21 or 22, further comprising the step of generating non-visual information when it is determined that at least one person has entered within the threshold distance of the client computing device.
Citation Information
Patent Citations
Detecting an attentive user for providing personalized content on a display
US20160110585A1
